Is there a way to throttle syslog

Hi

We use Log4J with its SyslogAppender to send messages to a central syslog-ng server, all running on Unix machines.

Is there a way (whether that's in Java or in Unix) to throttle the number of messages that are sent in order to avoid an upset server also upsetting the network?

The only option I can think of is to set the log level higher so less messages will actually be sent, but that isn't ideal as a important messages could be suppressed on a machine that is otherwise behaving itself.

I suppose in an ideal world a dynamically-changing level would be good: if the number of messages/second passes a certain level, the threshold rises, but at the same time that sounds a bit like overkill.

Any ideas?

Thanks

Rich

From serverfault Rich

• The only thing I know that it's close to that is a configuration to silent messages that are the same, like, if a log message repeats like 200 times you can ask syslog to log it once and ignore the others and only log a message saying that the message repeated 199 times more. Throttling the log can make you lose log messages and that's not desirable.

  Maybe you can put a QOS/traffic shaping between your server and the log server and use that to control the speed?

  From coredump

• I think the question is, do you care if you don't get all your logs on the central server? What you're talking about is essentially dropping messages--in which case, you'll lose logs. Is this okay? If it is, you've already answered your own question--raise the debugging level to only get messages you really care about.

  If, however, you're trying to match, say, a bandwidth constraint (such as Splunk's month processing limit), you'll need to write an intermediary server to take the logs from syslog and prioritize them. Its not difficult, but it is highly specific to your use case. One bonus with this method is that this middleman can immediately send important logs to the aggregation server and, at the end of the day/month, send the next highest priority logs that weren't sent originally. That way, you can fill the quota exactly.

  If you add more specific requirements (such as why you need to do this), and what you mean by limiting logs (duplicate lines? bandwidth? space? aggregation server can't keep up? etc..) then you'll get a much better answer.

  Good luck!

  Rich : Thanks for that - it's mainly a concern about network bandwidth.

  Redmumba : In that case, you may also want to investigate using compression, and sending your logs in chunks (just so the act of compressing it will give you more bang for your buck). :) And no problem, I'm happy to have helped!
  From Redmumba

• If you are using log4j correctly using a higher log level should not impact the import messages as they should have a high level. Log the detail locally with local rotation.

  From BillThor

• If it's bandwidth problem as you said, you could even use compression (It's supported on rsyslog, but i 'm not sure about syslog-ng). Also, if you are not using logs for real time alerts, you could make a script that read the local logs every some seconds and send a compressed aggregation of them.

  If you want a distributed solution then you can use the suggestion above. (Saves bandwidth on many links). But if you preffer something more centralised (With slightly more bandwidth waste, because you have to send useless logs to intermediate server) then Redmumba solution is much better.

  From Nikolaidis Fotis