I don't get what is so difficult about AMQP. These are clearly talented programmers, so what gives? Even if you're not a bank, simple features like message timeouts can make your infrastructure tremendously more resilient.
I'd chalk it up to the general benefits of eschewing needless complexity.
I can say, empirically, none of the many, many challenges we've had building and scaling Bump, have been related to Redis's capabilities as a messaging bus. So "good enough" wins again.
FWIW, I'd encourage you to still take a deeper look at AMQP, just because it includes features you may not know you need. While I can't pretend to know anything about your scaling challenges or the intimate details of how messaging is utilized, I can say that there are the lessons of deep experience with messaging baked into AMQP. You may have (or perhaps you already have) to implement some of these features in the future. I know that I gave AMQP the cold shoulder for a while, only to finally come around and find out it solved many of the frustrations we were facing.
Probably the less complexity also means a much bigger amount of messages per core.
With Redis in an entry level Linux box you can process 100k messages per second per core. I'm not sure if current AMQP systems can handle this amount of messages with commodity hardware.
Another reason why Redis can be a good approach I think is that it is a simple system: simple to understand, simple to deploy, very stable. There is support for Pub/Sub. It also supports many other data types, so for instance, want a priority queue? Do it with a sorted set. What some special semantic for your message passing? possibly with BRPOPLPUSH you can do it. And so forth.
A common case of this flexibility is shown by RPOPLPUSH (without the "B" so the non blocking version). Using this command with the same source and destination list will "rotate' the list providing a different element to every consumer, but the data will remain on the list. At the same time producers can push against the list. This has interesting real world applications when things must be processed against and again (fetching RSS feeds, for instance).
So Redis is a pretty flexible tool for messaging, and I think there are for sure great use cases for AMQP but with big overlaps with Redis, and also with use cases where Redis is definitely a better alternative.
Given the same semantics, RabbitMQ pushes 100k+ per core as well. It utilizes multi-core machines quite effectively too. Obviously as you tune nobs for more durability or reliability, you're going to take a performance hit. IMHO, Redis is a good tool for messaging if you already use Redis in a big way and don't need reliable delivery or any of the advanced AMQP routing. In sort of the same way a table in MySQL can be used as a queue if you already have a MySQL server and you're only going to be pushing a few hundred messages per minute.
I call BS on rabbit pushing 100K messages per core and I really like rabbit. But it is no where near 100k messages/sec per core.
AMQP does have extra capabilities and is a good messaging system and has advanced routing features, but you need to learn what exchanges, queues and bindings are and how they relate before it is useful.
I've used rabbit in production on multiple systems and it is still running on some of those. But I have switched to redis for most of my messaging needs because of the built in data structures and persistence. It makes it a much more versatile server and it is much easier to admin and much more stable. Rabbit is too easy to push over when you run it out of memory.
But I had to chime in and refute your 100k messages per core on rabbit. 20k maybe with the java client, more like 5-7k with a ruby or python client.
I can still get 80k/sec with a ruby client on redis.
The two servers are very different, redis is a swiss army knife of data structures over the network, that is why it is so useful. AMQP and rabbit are targeted more at enterprise messaging and integration where raw speed doesn't matter as much as complex hierarchies of brokers and middleware.
Given the same semantics, RabbitMQ pushes 100k+ per core as well.
Until it grinds to a halt for no obvious reason</snark>
We've dropped RabbitMQ on a project a while ago not for performance but for stability and opacity reasons.
I went into detail about the issues in a post a few months ago so I'll skip that here. My take-home is that unless you have complex routing requirements that can only be realized with AMQP then you should think long and hard if you really want the aircraft carrier of a component that is RabbitMQ in your dependency list.
Redis/Resque, beanstalk and Celery cover the overwhelming range of use-cases just fine and are complete no-brainers to operate in contrast to wrestling with AMQP topologies, flakey client libraries and an opaque (to most) erlang blackbox.
