These platforms enable horizontal scaling, but ironically, many apps only need to scale horizontally because the base instances offer such underwhelming specs.
It's frustrating to see engineering teams in 2024 spending countless hours optimizing their applications to run on what essentially amounts to 2008-era hardware configurations.
Worked on serverless computing for several years at a major cloud provider and think I could offer a major reason for this: many customers (and us; spent the majority of my time working on projects motivated by a desire to increase throughput per infra $) found it challenging to implement software capable of reliably handling high levels of concurrency.
Start from the premise that the majority of developers out there have no experience dealing with concurrency, then consider that changing something's approach to concurrency/parallelism can occupy several developers' time for months or possibly years. Then consider that for a business the absolute costs of using 2-20x more instances than "necessary" may not justify the investment to actually switch to that. Then it makes sense why people so often choose to use concurrency = 1 (in which case you basically want the smallest instance size capable of running your app performantly) or whatever concurrency setting they've been using for ages, even if there are theoretical cost savings.
This is one reason AWS Fargate is so good. No, it isn’t developer friendly, but it gives you performance that actually matches the CPU allocation while still abstracting the VMs.
It doesn’t, actually, because you have zero guarantees on what generation of hardware you’ll end up on from deployment to deployment. Haswell this time, Ice Lake next time. What fun!
It's frustrating to see engineering teams in 2024 spending countless hours optimizing their applications to run on what essentially amounts to 2008-era hardware configurations.