on one of our machines that sends generates logs we do around 2400/s when the application is under heavy use. we have 9 machines that generates logs but they all generate different amounts. we mostly are using heka for generating stats from log files because we are too lazy to instrument the code and we have excellently detailed and formatted logs :) but we do have some logstash machines still pushing stuff to elastic search for low traffic applications we run.
we found that when using logstash even just for pushing stats to statsd it was not performing well enough. i've experimented with hekad pushing to elasticsearch on our staging cluster and performed well enough but we had weird problems showing up in nagios when we were using logstash+elasticsearch in production (checks were timing out even though we were seeing no degradation of performance on the servers). because of this it is quite difficult to get any kind of central log pushing into production. :(
we found that when using logstash even just for pushing stats to statsd it was not performing well enough. i've experimented with hekad pushing to elasticsearch on our staging cluster and performed well enough but we had weird problems showing up in nagios when we were using logstash+elasticsearch in production (checks were timing out even though we were seeing no degradation of performance on the servers). because of this it is quite difficult to get any kind of central log pushing into production. :(