I recently had to carry an evaluation of centralized logging facilities for my $dayjob. A few days later that included some experimentation (and some frustration) with LinuI managed to get a setup into place that emulated 8 syslog clients pushing messages to the centralized syslog server. Thus, with the help of the excellent loggen utility that’s part of syslog-ng, I opted to benchmark rsyslog.
A few days later here are some interesting findings:
- UDP stands for Unreliable Datagram Protocol: sure you can add some app-level logic to get around the limitations of UDP but syslog doesn’t have any. In my experiments messages started being lost at as low as 3,000 messages per second (mind you this was on a dedicated network backed by a 10GbE switch and links)
- Unix sockets (the /dev/log device) are reliable but don’t scale as well as a TCP listening socket
- Rsyslog rulesets are your friend. Just remember to create a separate queue per ruleset so that you take advantage of the multiple threads that your CPU will probably support
- Logging to an HDD introduces a healthy performance penalty. Make sure that you try out logging to the “/dev/null” device to see how much you could gain by reducing the impact of the disk I/O by moving to a RAID-1+0 or even a network storage facility (NFS, iSCSI LUNs etc). Trying to optimize your filesystem doesn’t hurt either.
- When every last bit of performance has been squeezed, check out if you’ve already tried increasing OMFileIOBufferSize 128k and MainMsgQueueSize to 100k.
- Know your limits: there is no point in storing your logs in a database if your database can take 4000 writes per second but your expectations are for 8000 new messages per second. Buffering can help but only that much.
- KISS: Keep it simple. rsyslog (and syslog-ng for that matter) is extremely powerful but all those nice features may hurt performance. If you have to implement complex rules and filters and (…) make sure that you figure out the best and most efficient way to do so. Once you have, go back to step (6) and see if it’s efficient enough.
- Plan ahead: just because it works for now doesn’t mean it’s going to work in one month or one year.
Making sure that I follow the above advice my centralized logging server scales to 250,000 messages per second, using 512 bytes long messages. This is enough to kill a Gigabit link, so it’s not too bad 🙂 Hardware for the centralized log server was an HP BL460c blade with a single X5550 processors and 6 SAS 15K 146GB hard disk drives, configured in a RAID-0 fashion using the built-in RAID controller.
Wrapping up this effort I shortly evaluated syslog-ng and found its performance to be notably worse. With minimal tuning I could barely exceed 50,000 messages per second. And given its creators claims for 75,000 messages per second I doubt it could scale anywhere near to the levels I was able to achieve with rsyslog.