I’ve seen teams destroy themselves by solving complex problems that didn’t need to be solved. Instead of solving the real problems, they imagined a problem that was more interesting and complex, and tried to solve that.
in my defense I can think of at least four times I needed to prototype a complex solution to understand why it was the wrong solution and then discard it
+100. A lot of us love to over-engineer things because that's more fun to solve and looks good on a resume / evaluation.
Kubernetes, (micro-)services, queues, nosql, lambda and other aws service are examples that I've seen in the wild. These can absolutely be valuable and cost-effective, if the problem needs that complexity.
I love engineers that focus on the user, problem and business. They are soo valuable, especially for startups and smaller companies. Engineers that love solving complex problems without thinking much about the user are still awesome, if they can operate in an organisation where that critical thinking is done by others (product people or other engineers)
I once wasted several months of schedule and payroll wrestling to improve a DNN model - which resulted in that model getting much much better thanks to tons
of creative and interesting work.
We swapped the learned model for a perfect a oracle and realized the actual problem
was elsewhere in the system and was easily fixed. None of those model enhancements were needed.
Not the person you asked, but working on remote sensing and perception I have seen (not once) the teams turning from improving use of available sensor data to things like cutting edge messaging technologies to support real-time data delivery from thousands of sensors (when they have three sensors today and might get another one next year and their current simple streaming will work just fine for up to 20).
While those are interesting problems they have nothing to do with their current perception challenges (that have to be solved for a successful deployment), steal resources from the team and make life harder for a few remaining people actually working to solve current challenges, as they now need to switch to a more complex messaging and rewrite some of their existing utilities.
These weren't fatal to the project or team, but did contribute extra work for net negative value (slower & more maintenance, with no relevant extra features):
- Using Kubernetes to host an application when Docker Compose would do (specialist application with <10 concurrent users).
- Using DynamoDB to store a few kB of temporary data in a processing pipeline, despite having several existing PostgreSQL instances already and no DynamoDB ones.
I worked on a team for a short while that decided to use Cassandra to store a few hundred thousand accounts, despite the data changing only slowly and being an exact match for a relational database. It was one of the worst choices I've ever seen in my career, and the only reason I could find was that people at the time wanted to learn Cassandra.
Sounds somewhat familiar. I worked at a place that replaced a basic LAMP stack with microservices written in Perl and MongoDB as a backing store, for the sole purpose of raising complexity.
And left Mongo on the default settings for the time (speed of return over reliability of saving data), so they ended up with a reporting replica which was MySQL.
Don't think I've seen anything before or since which was architected so exactly backwards.
Think my company burned a year on a solution using Cassandra. I think the problem was 1/3 Cassandra and 2/3rds the people that would pick Cassandra amd Java over PostgreSQL and C# or go.
Years ago, my team at AWS considered writing our own declarative language to configure EC2 instances which would replace some old functionality within CloudFormation (cfn-init) until I pointed out you could achieve the same thing with our existing features (State Manager Ansible playbooks)
I worked at a place where they finally ditched their Hadoop project after a couple of years and who knows how much money. Only to implement a big Kubernetes cluster instead, where they four years later have burned through three different commercial Kubernetes distributions.
