Startups? I work for a finance firm, and while we certainly have a need for large farms of servers to store data, my current team keep talking about web request latencies as an important infrastructure concern when the literal maximum number of users is in the tens of thousands.

Which is, you know, 1 maybe 2 machines with nginx plus 1 for redundancy. Our internal services are slow because people cohost them with batch jobs, and no other reason.

It's so frustrating, because when it all boils down to it, the story is really quite simple:

1) What metrics reflect your customer experience? Monitor them, alarm on them. Target them as a priority for improvement. Check in on them weekly at a minimum.

2) What metrics make up the metrics that reflect the customer experience? Monitor them as well, and consider whether alarming on them is the right thing to do. Use these to direct what you target to solve 1.

3) What do you need for SPOF resilience? If you've got something critical, you need a minimum of two servers running it, and no more than, say, 40% total CPU usage so that you've got sufficient overhead to cope with a single host failure and any unexpected work increase.

It isn't just startups. And I agree with another commenter: it seems like resume driven development.

There's also a weird peer-pressure involved. Overheard a conversation a while back that summarizes it nicely - someone was talking about a scheduling system written they've used for years, and mentioned it was written in Perl. Another participant guffawed, and after the requisite Perl-bashing, the original person allowed that, yes, even though it worked fine, they should rewrite it.

No idea what company that was, but I'd love to work in a place where that was the most pressing concern on my plate.

I've encountered this sort of thing too many times here in the last few years. A set of Python scripts to ETL a few hundred MB of data once daily is suddenly desperately in need of replacement with a real-time, fault-tolerant MQ backed server (I've seen more than one of these, from custom ZMQ proposals to Kafka). A set of two or three scripts run on a compute server that are kicked off by cron and that do some work in batch on a modest set of data (few hundred GB) suddenly must be replaced with an asynchronous task management service capable of managing a huge set of processes with complicated dependencies (Jenkins, Luigi). A bit of data one team produces (once daily) has another team asking for it: instead of exporting the data from the DB directly or standing up an endpoint on, say, a Flask server there's a drive to slap a Spring microservice architecture in place. And so on. It's really a sight to behold, the rationalizations people go through who, in other contexts, laugh at the inferior logical reasoning abilities of non-CS folks.

Well...there's a lot of context in "works fine," from "it is perfect and there's nothing to add to it" through "good enough" to "it sort of works within most expected inputs, but everyone is afraid to touch it, so we've built a bunch of hacks around it."

Were all of these people competent?

Honest question. I have only seen the "it's written in X, so therefore it must be re-written in something nicer even though it is working" thinking from incompetent people who were just trying to take ownership of something they didn't quite fully understand. Though I never have seen it in a appropriately functioning commercial setting; if management is competent, they'll immediately recognize the high costs with no concrete benefit and say no.

It's one thing to say "we have to re-write this because it uses Java applets, and Java applets are problematic because Oracle is dropping support for them, so our customers are going to be screwed soon if we don't do something." It's another thing to say "we have to re-write this because it's in Perl because Perl is something I don't like."

I've seen this situation multiple times, and yes the developers involved were competent. They were even well-meaning, and wanted to build something for the benefit of the company, not just their resumes.

I think the tendency to over-engineer and over-polish comes mostly from getting too invested in one particular project or task. The developers have "professional pride" - they want to deliver software that has good architecture, high test coverage, easy to understand and maintain code, reliable, scalable, etc.

This means competent developers are very tempted to continue working on a project as long as there are possible improvements to it, even if these improvements do not make business sense. Nobody wants to admit that "cron job that fails once per month" is a sufficient solution when they can see a better solution, and go work on the next hacky cron job instead.

To your last line -- I've got a PHP sub-system that I would love to have re-written to match our chosen stack language set, maybe simply because we could then have fewer required skills on the team (few of us are PHP guys). But for all its warts, it works. And that matters. I am thus hesitant to re-write for no other reason than 'we hate it'. It would be a lot of work for zero functional gain. (All benefits would be non-functional - and that stuff gets put on the back burner)

OK, but the other side of that is all of us looking at resumes and going "wow, that's cool that this person built that" instead of "that sounds overengineered and silly".

Well...

1. This is why you ask in the interview about it. 2. I don't know about you, but if I see certain buzzwords, I assume that the candidate merely chases the latest fads and subtract points accordingly until counter-evidence is produced.

Who is "us"? These days if I see things like that on a resume I'm more likely to be skeptical than awed.

The point is the resume building is done for the purpose of impressing the people reading the resume. If it didn't have that effect then nobody would bother resume building.

You're not using Rails, aren't you? What stack do you use to handle 1000s of requests per second on a 512MB VPS?

I don't know about the specific overhead of rails. But if you had a uniform distribution of 2000 requests a second, and each request took up to 50ms, then you'd only have 100 requests at any given time. So each request could take up to 5mb, which is generous unless you're doing something explicitly fat.

5 ms isn't generous at all for a backend.

You mean 5 mb? I can't say I have a ton of experience. But of the three projects I've worked on, the fattest was a Vaadin CRUD app, which stores the user session/state on the server and it could be around 5mb. More context: This isn't counting any DB effort.