Here's my thought on the current state of the industry. DevOps at some point was not a specialty that you hired for, it was a way of thinking about your team's responsibility. Your team would make an application and your team would run that in production. If you wanted to test things before deploying, you would do that. If you wanted automated deploys, you would set that up. No middleman with competing concerns between you and your users.
Eventually, people had a hard time finding well-rounded individuals that could design, develop, test, and deploy software. It seems to be a rare skillset, and people are resigned to not being able to hire for that kind of role. So, all of these ancillary concerns got split off into separate teams. You have a design team, a software engineering team, a test engineering team, operations, and so on. DevOps changed from "developers that operate their software" to "developer operations", which is just your 1990s operations team with a new name. You the developer want something, it goes on a backlog for some other team, you wait 6-8 years, you get your thing.
All the complexity of the devops world comes from having one team writing the software and one team running the software. An example are service meshes. They are super popular right now, and everyone and their mother is writing one and selling it for tens of thousands of dollars per year. To the software engineer, having two applications communicate over TLS is pretty simple; you read the certificates and keys from disk or an environment variable, throw them into a tls.Config, and give that tls.Config to your internal servers and internal clients. But, what happens in the real world is that the organization says something like "all apps must use mTLS by January 2023". The software team says "meh, we don't care, we'll get to it when we get to it". So the poor devops team is stuck figuring out some way to make it work. The end result is a Kubernetes admission controller that injects sidecars into every deployment, which provision TLS keys from a central server at application startup time. The sidecars then adjust iptables rules so that all outgoing connections from the original application go through the proxy, and if some distributed policy says that the connection is supposed to be mTLS, it makes that happen. Basically, because nobody on the dev team was willing to spend 15 minutes learning how to make this all work, it got bolted on by $100k worth of consultants, all for a worse result than just typing in a very small number of lines of code by yourself. That's the state of devops. The people writing the software won't run it, so you have to add complexity to get what the organization wants.
I think it's terrible, but that's the fundamental disconnect. When you need to change how software works without being able to edit the code, the workarounds get increasingly complicated.
As always, what looks like a software complexity problem is actually an organizational complexity problem. The way I've managed this in the past is to organize a lot of knowledge sharing, and make a conscious effort to avoid hiring too many specialists. At my current job my team used to make a SaaS product, and our team was a combination of backend software engineers, frontend software engineers, and some engineers with experience with operations. We were one team; frontend engineers would write Go code, backend engineers would make React changes, and we all did operational drills ("game days") once a week. The result was a very well-rounded team. Everyone could deploy to production. Everyone could be on call. Everyone could fix problems outside of their official area of expertise. I wouldn't have it any other way. The industry, however, deeply disagrees with that approach. So you're going to have testing teams, devops teams, etc.
Eventually, people had a hard time finding well-rounded individuals that could design, develop, test, and deploy software. It seems to be a rare skillset, and people are resigned to not being able to hire for that kind of role. So, all of these ancillary concerns got split off into separate teams. You have a design team, a software engineering team, a test engineering team, operations, and so on. DevOps changed from "developers that operate their software" to "developer operations", which is just your 1990s operations team with a new name. You the developer want something, it goes on a backlog for some other team, you wait 6-8 years, you get your thing.
All the complexity of the devops world comes from having one team writing the software and one team running the software. An example are service meshes. They are super popular right now, and everyone and their mother is writing one and selling it for tens of thousands of dollars per year. To the software engineer, having two applications communicate over TLS is pretty simple; you read the certificates and keys from disk or an environment variable, throw them into a tls.Config, and give that tls.Config to your internal servers and internal clients. But, what happens in the real world is that the organization says something like "all apps must use mTLS by January 2023". The software team says "meh, we don't care, we'll get to it when we get to it". So the poor devops team is stuck figuring out some way to make it work. The end result is a Kubernetes admission controller that injects sidecars into every deployment, which provision TLS keys from a central server at application startup time. The sidecars then adjust iptables rules so that all outgoing connections from the original application go through the proxy, and if some distributed policy says that the connection is supposed to be mTLS, it makes that happen. Basically, because nobody on the dev team was willing to spend 15 minutes learning how to make this all work, it got bolted on by $100k worth of consultants, all for a worse result than just typing in a very small number of lines of code by yourself. That's the state of devops. The people writing the software won't run it, so you have to add complexity to get what the organization wants.
I think it's terrible, but that's the fundamental disconnect. When you need to change how software works without being able to edit the code, the workarounds get increasingly complicated.
As always, what looks like a software complexity problem is actually an organizational complexity problem. The way I've managed this in the past is to organize a lot of knowledge sharing, and make a conscious effort to avoid hiring too many specialists. At my current job my team used to make a SaaS product, and our team was a combination of backend software engineers, frontend software engineers, and some engineers with experience with operations. We were one team; frontend engineers would write Go code, backend engineers would make React changes, and we all did operational drills ("game days") once a week. The result was a very well-rounded team. Everyone could deploy to production. Everyone could be on call. Everyone could fix problems outside of their official area of expertise. I wouldn't have it any other way. The industry, however, deeply disagrees with that approach. So you're going to have testing teams, devops teams, etc.