My, perhaps overly cynical view, is that Message Queue architecture and blogging was all about "Resume Driven Development" - where almost everybody doing it was unlikely to ever need to scale past what a simple monolith could support running on a single laptop. All the same people who were building nightmare micro service disasters requiring tens of thousand of dollars a month of AWS services.
These days all those people who prioritise career building technical feats over solving actual business problems in pragmatic ways - they're all hyping and blogging about AI, with similar results for the companies they (allegedly) are working for: https://www.theregister.com/2024/06/12/survey_ai_projects/
I'm sure this happens. But ... most websites I load up have like a dozen things trying to gather data, whether for tracking, visitor analytics, observability, etc. Every time I view a page, multiple new unimportant messages are being sent out, and presumably processed asynchronously. Every time I order something, after I get the order confirmation page, I get an email and possibly a text message, both of which should presumably happen asynchronously, and possibly passing through the hands of more than one SaaS product en route. So given what seems to be the large volume of async messages, in varying/spiking volumes, possibly interacting with 3rd party services which will sometimes experience outages ... I gotta expect that a bunch of these systems are solving "actual business problems" of separating out work that can be done later/elsewhere, can fail and be retried without causing disruptions, etc in order to ensure the work that must happen immediately is protected.
Bingo - I work on the backend of a medical system and basically anything that interacts with a 3rd party gets put into a queue so our application doesn't choke immediately when one of them has issues. We also have some uses for it within our system.
As far as the question, I was thinking that queues have probably just become a standard aspect of modern distributed systems; it's considered a pretty foundational cloud service for any provider (though we just run RabbitMQ ourselves and it has worked well for us).
At work we usually integrated with a queue, and then some partners/customers wanted a synchronous flow because the user has to see and pick from data we don't own, and now life is pain.
Tracking, visitor analytics, and observability type things are all (in general) going out to 3rd party specialist services for those things, and getting dropped into a time series database (or, for us old school gray beards, a log file) and processed later. It's rare for the website devs to be doing anything more complex that adding in even more javascript to their pages for those, no need to message queues for that.
Order confirmation emails and sms messages are triggered by the order submission, and again usually sent off to a 3rd party bulk email or SMS service. Twilio or Campaign Monitor or Mailchimp will have queues and retry mechanisms, but again the website devs are just firing off an API call to some 3rd party that's dealing with that.
So there are no doubt message queues being used in non-sexy 3rd party services, but those companies probably consider that kind of thing to be their "secret sauce" and don't blog about it.
> Order confirmation emails and sms messages are triggered by the order submission, and again usually sent off to a 3rd party bulk email or SMS service. Twilio or Campaign Monitor or Mailchimp will have queues and retry mechanisms, but again the website devs are just firing off an API call to some 3rd party that's dealing with that.
In my case, I need to compile templates for the e-mails to be sent, which is somewhat slow. Even if I have an in memory cache for the compiled templates that can then be quickly filled in with actual data, I don't want to make the first user to request them after a restart/flush wait like 4-7 extra seconds upon clicking on the confirm order button (in addition to any other 3rd party calls, checking payment status etc.).
Ergo, I need to carry the actual e-mail preparation logic (regardless of whether I send it myself, or use a 3rd party service) out into a separate thread. The problem then is that I most likely don't want to create a new thread for every e-mail to be processed, so I need a queue for the requests and one or multiple worker threads. There is functionality for this in the stack I'm using so no big deal, except I can't pass live DB entities across threads, so I also need to serialize the data before passing it off to the queue and deserialize it inside of the queue (or just pass some IDs and do DB calls within the queue).
Essentially I've created a simple queue system in the app code, since processing that data definitely shouldn't make the user wait on it. I can see why some might also go the extra step and opt for something like RabbitMQ, since at the end of the day my queue system is likely to be way more barebones than something that's been around for years. But until a certain point, YAGNI.
Though to be honest, if it wasn't template compilation, it might as well be slow DB queries, slow external service calls, or anything else that you wouldn't want to do in the lifecycle of the user's request, if it concerns something that should happen in the background, not even related to a particular stack.
There's always a queue somewhere at some level of resilience. Sometimes it's as mundane as the TCP buffer, some other times the other end points may become unresponsive for a while, including the time series DB you seem to want to take as the source of truth, or whatever that tries to resend and reconcile the requests.
One word: scale. The services you mention above do require scale if commercial. OP argues and I somewhat agree that lots of resume driven tech was oversold and overused making things more complicated and expensive than they should have. Once tech gets more mature it’s harder do misuse and it is used where real needs arise.
This is true, but in my opinion badly misunderstood.
There are a huge number of "commercial" things that are hitting several million dollars a month in revenue running Ruby On Rails or WordPress/PHP. You can scale a long long way with "boring technology".
Way too many people think that are "the unicorn" who's user base is going to grow so quickly that then need billion simultaneous user scale right now - instead of realising that before they get even close to that they'll be generating enough revenue to have an engineering team of hundreds who will have rewritten everything two or three times with more suitable architectures already.
If you need a billion users to turn a profit, then whether you admit it or not your business model is "burn VC money until they stop giving it to us or we pivot using Doctorow's Enshittification blog posts as a guide book". That though, is a vanishingly small percentage of all dev work. Most business models have a way too make real profits of thousands or perhaps tens or hundreds of thousands of transactions a month - and they should be rolling in profit to reinvest in future higher scale development well before they run out of fairly pedestrian "boring technology" platforms. Horizontally scalable Java/PHP/Ruby/Node with vertically scaling databases on you cloud provider of choice is a well known and battle tested way to generate real value and cashflow in probably 99% of all businesses.
I absolutely agree. But keep in mind that a lot of these services are made by startups that want to be acquired by someone with big pockets. Using hyped tech helps the sale. Don’t ask me why…
> My, perhaps overly cynical view, is that Message Queue architecture and blogging was all about "Resume Driven Development" - where almost everybody doing it was unlikely to ever need to scale past what a simple monolith could support running on a single laptop. All the same people who were building nightmare micro service disasters requiring tens of thousand of dollars a month of AWS services.
Yes, that is cynical. People have been building architectures off MQ for a much longer time than microservices have been around. Lots of corporates have used JMS for a long time now.
Anything can be 'resume driven design'. If someone is rewarded with a raise, promotion, or even just praise sometimes, for applying a technology to a problem without being required to prove if the technology is appropriate they'll find a way to jam that tech into their domain regardless.
Sometimes that promotion is rewarded by going a different company while being able to say "yes, I used X in my previous role."
I have also seen a lot of cases where engineers would use the more unnecessarily complex structure on purpose to make themselves less replaceable, as it would take longer time for newcomers to get familiar with the environment deployed.
The reality is that management never cares how non-replaceable an engineer is, fires them anyway, a bunch of stuff breaks and the newcomers are stuck holding the bag.
These days all those people who prioritise career building technical feats over solving actual business problems in pragmatic ways - they're all hyping and blogging about AI, with similar results for the companies they (allegedly) are working for: https://www.theregister.com/2024/06/12/survey_ai_projects/