rsyslog evaluation

I recently had to carry an evaluation of centralized logging facilities for my $dayjob. A few days later that included some experimentation (and some frustration) with LinuI managed to get a setup into place that emulated 8 syslog clients pushing messages to the centralized syslog server. Thus, with the help of the excellent loggen utility that’s part of syslog-ng, I opted to benchmark rsyslog.

A few days later here are some interesting findings:

  1. UDP stands for Unreliable Datagram Protocol: sure you can add some app-level logic to get around the limitations of UDP but syslog doesn’t have any. In my experiments messages started being lost at as low as 3,000 messages per second (mind you this was on a dedicated network backed by a 10GbE switch and links)
  2. Unix sockets (the /dev/log device) are reliable but don’t scale as well as a TCP listening socket
  3. Rsyslog rulesets are your friend. Just remember to create a separate queue per ruleset so that you take advantage of the multiple threads that your CPU will probably support
  4. Logging to an HDD introduces a healthy performance penalty. Make sure that you try out logging to the “/dev/null” device to see how much you could gain by reducing the impact of the disk I/O by moving to a RAID-1+0 or even a network storage facility (NFS, iSCSI LUNs etc). Trying to optimize your filesystem doesn’t hurt either.
  5. When every last bit of performance has been squeezed, check out if you’ve already tried increasing OMFileIOBufferSize 128k and MainMsgQueueSize to 100k.
  6. Know your limits: there is no point in storing your logs in a database if your database can take 4000 writes per second but your expectations are for 8000 new messages per second. Buffering can help but only that much.
  7. KISS: Keep it simple. rsyslog (and syslog-ng for that matter) is extremely powerful but all those nice features may hurt performance. If you have to implement complex rules and filters and (…) make sure that you figure out the best and most efficient way to do so. Once you have, go back to step (6) and see if it’s efficient enough.
  8. Plan ahead: just because it works for now doesn’t mean it’s going to work in one month or one year.

Making sure that I follow the above advice my centralized logging server scales to 250,000 messages per second, using 512 bytes long messages. This is enough to kill a Gigabit link, so it’s not too bad 🙂 Hardware for the centralized log server was an HP BL460c blade with a single X5550 processors and 6 SAS 15K 146GB hard disk drives, configured in a RAID-0 fashion using the built-in RAID controller.

Wrapping up this effort I shortly evaluated syslog-ng and found its performance to be notably worse. With minimal tuning I could barely exceed 50,000 messages per second. And given its creators claims for 75,000 messages per second I doubt it could scale anywhere near to the levels I was able to achieve with rsyslog.

Advertisements

Tags:

18 Responses to “rsyslog evaluation”

  1. Michael Iatrou Says:

    Excellent summary, thanks for sharing! On which exactly OS/file-system did you do the evaluation?

  2. mperedim Says:

    I thought the “optimize your filesystem” link would give out part of this answer.

    Tests were done on CentOS 5.4, ext3. Note that you don’t have many options here, just linux and *BSD [ see http://www.rsyslog.com/doc-rsyslog_packages.html ].

  3. Tweets that mention rsyslog evaluation « ~mperedim/weblog -- Topsy.com Says:

    […] This post was mentioned on Twitter by Yiorgos Adamopoulos, mperedim. mperedim said: short blog post: rsyslog performance evaluation (and how to achieve a 250,000 messages per second rate) – http://is.gd/6KUov […]

  4. Sotiris Tsimbonis Says:

    Which version of rsyslog ?

  5. mperedim Says:

    My evaluation was intended for a longer term project so I could afford the risk and go bleeding edge (5.5.1).

  6. David Lang Says:

    I have measured rsyslog over gig-E UDP at >300K messages/sec with no data loss (dedicated switch) for bursts that did not overflow the ram (up to 1M messages on one very high-memory box). I was able to get >75K messages/sec to disk (if that’s all that it was doing with the logs)

    so your performance numbers are sounding low to me. It’s very possible that there are config differences that account for this.

    one of the key things for me was disabling DNS lookups.

    • mperedim Says:

      I have measured rsyslog over gig-E UDP at >300K messages/sec with no data loss

      Perhaps it was something in my setup making UDP a non-starter or perhaps it was the fact that I used an unstable beta; it was definitely not the switch (these were HP blades connected over a VC-10 module :)). TCP gave really good performance straight away, so I admit I didn’t investigate this further. YMMV.

      so your performance numbers are sounding low to me.

      The 250K messages per second quoted in the original post was actually logging messages to the disk. Given your feedback for >75K I guess it’s OK.

      It’s very possible that there are config differences that account for this.

      Definitely. Note that in any event the post is intended to provide some general guidelines and set capacity expectations from an order of magnitude perspective [the numbers will always defer depending on the actual setup]. To put it otherwise, if you get 300K instead of 250K you probably tried the same tricks I did. If you got 500-600K though … please write a trackback entry explaining your setup and how you did so 🙂

      • David Lang Says:

        I only had 1G ethernet, which was basicly saturated at that message volume. I was using tcpreplay to send the messages and was also at the limit of how fast it could send the packets.

        in the last few days there has been a lot of work in speeding up the output side of rsyslog, so give it a little time to work the bugs out and I think you will see significant improvements

  7. mperedim Says:

    so give it a little time to work the bugs out and I think you will see significant improvements

    I am sure they will. If nothing else the epilogue in my post, comparing it against syslog-ng, is a praise to rsyslog’s capacity and scalability, and the reason that we went with rsyslog for the centralized logging needs of the project I briefly mentioned above 🙂

  8. sh-beta Says:

    Can you post the configuration you used in these tests?

  9. Peter Czanik Says:

    Could you post your rsyslog.conf? Or send me in private? I did a few tests on a core2duo with rsyslog and syslog-ng and with stock config (openSUSE 11.3) syslog-ng was about 4 times faster. After doing some rsyslog tuning based on your blog, I still reached only a little bit more than half of syslog-ng’s speed.
    ~62k for syslog-ng
    ~18k for rsyslog
    ~32k for rsyslog using tuning tips from this page

    • mperedim Says:

      Hi Peter/sh-beta,

      Please note that my tests were with a much more powerful system (2x X5570).

      The configuration by itself is not worth enough without a couple of extra notes. I’ll do a new post with a trackback to this one within a couple of days.

      • Mahesh Says:

        Hi,
        could you also please share your rsyslog.conf file to me?
        I can barely bring the config to get 100 messages per second
        with a backend of mysql (rsyslog + mysql)
        thanks
        Mahesh

      • mperedim Says:

        This really sounds like a bottleneck on mysql. A quick test you can do is send the rsyslog output on /dev/null, this should eliminate most of the overhead related to writing the logs, taking away items such as disk speed or mysql overhead.

        Frankly I didn’t test the mysql component at all so I am not that certain what kind of performance level you should expect out of it.

        P.S. I’ll see if I can do a follow up post to share the configuration. Not really certain how useful it could be in light of the rsyslog developments over the 3+ years since this post was published.

  10. Nima0102 Says:

    Thanks a lot for sharing.
    Do you have any benchmark for recently version of rsyslog?

  11. mperedim Says:

    No I don’t, since 250,000 messages per second was more than enough for me (at least an order of magnitude more than what we needed). I do seem to recall though that the rsyslog author is confident that newer versions should perform much better.

  12. Access Log 저장 | itconsultant Says:

    […] rsyslog evaluation […]

  13. davidelang Says:

    even more important is the developments in the last few months. As of rsyslog 7.3.5 it’s able to insert multiple messages in a single transaction.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: