It's business-generated complexity, not complexity out of necessity. And I don't mean business made the complexity directly, but it's an indirect effect of trying to commodify developer jobs.
If you have a job posting describing "work on our website" you get all sorts of random candidates that have no real place in your company, but when you say "developer with framework XYZ experience" you can skip everything, judge their framework experience and move on (or so it is thought to work).
This means that to be attractive you need to use it as a developer, but inside a company this stuff has to be used to make sure that the current developer(s) can be replaced easily (not how it realistically works beyond CRUD apps), and expanding the workforce is easier because attracting developers with technology keywords is easier than business job descriptions.
End result: using technology not because it's the best fit, but because it is off-the-shelf. Essentially a modern version of using Oracle and Windows NT 3.51 back in the day, you're using it because it's an easy off-the-shelf keyword, not because it actually has anything to do with what you are technically trying to get done. Enterprise IT and HR shaking hands all over again.
Some more expansion on this topic (story time):
Generally when you are large enough as a team or business unit to have lots of collaboration, you need some way to exchange information and ideas, and work together using similar-ish principles. That can mean various things, from all using the same OS and IDE to having up-to-date API documentation and integration tests. This means that if you have to transport some idea, project or application across people (and teams) you need a method and format to do so.
This nearly automatically also means you can't do solo yolo all day and you need to have rules and packaging standards so you don't end up figuring out what Pete did on his machine so it works on Jack's machine but not Jane's. Chroot-in-a-tarball works, virtual machine images, maybe autotools with a bunch of setup scripts. But most of the older methods all have the same problem: your systems all need to be rather similar to work with it, and you need to have systems knowledge to make use of it.
With the expansion of the workforce, that becomes more and more scarce, to the point where most Junior and Medior engineers have no clue how their computer actually works, how to manage an operating system, how how to understand the influence of externalities (be it them influencing other teams, or the other way around). People with less experience are easier to 'create' (yes, I know, we're not actuallt manufacturing people on-demand for work), as it requires less affinity, less time, and less education resources. So we can get more of them, and as long as they are put in an environment where they don't need to be as capable as 'full fat' engineers, the business can keep running.
So, what do we do? We package and standardise. First with environments, we have virtual machines, images, packer, and can see those local virtual machines with ansible and saltstack and if it works there, the project will work on other workstations and servers just the same, since it was all provisioned the same. No more guessing. But this is a really heavy method of doing this, and the greybeards will complain that just using some file protocol to directly edit something or a simple jail or chroot was much lighter (faster, less resources etc). At the same time, from a security perspective, sandboxing stuff gets more traction, and with namespaces and control groups we also get docker (pre-OCI), which gets a bunch of people excited because you can now make transportable environments that are essentially jails, but without having to know anything about jails or bsd and it works on Linux!
Parallel to this, we get various tools for dependency management, mainly derived from the same concepts that drive dpkg/apt and rpm/yum, because it worked so well for system packages, why not use it for nearly every language under the sun? This gets into DLL-hell rather quickly, contaminates the operating system by installing packages needed for your projects globally for the entire system, so now whenever you switch to a different projects you basically have to reinstall. Boo! But luckily someone heard of this fancy new container thing and you can just do all of that stuff in there!
Now we can simply create those project environments as container images (got some real traction once OCI was a thing), so an engineer working on some new feature has even less of a requirement regarding their skillset. Just hire someone who can run docker containers and work on framework XYZ! Tons of those available. So instead of having teams with 5 highly skilled workers with deep knowledge and a high cost, you just get 3 Juniors for bulk work, 1 medior for checking their work and 1 senior that's in that classic "know as much as possible" bucket do mitigate any complex issues that may arise.
Fast forward to today, the new greybeards are the senior engineers of a decade ago, less and less new people with deep knowledge have been added to the workforce, but large masses of either highly specialised or shallowly trained people are being hired. They are not unskilled, but just skilled for a specific piece of the business, because that works for other departments so why not IT? And there is a shortage after all...
The now rare and in high-demand deep knowledge engineers are tired of getting the same questions for the same architectural problems and same business complexities, so they come up with software frameworks that they can apply to the problem. It doesn't matter if the other team members have a varying gradient of understanding of the details of the solution, as long as the tickets get closed and added value is perceived by non-technical people. Those same people may see those framework names, and recognise them in various important places around the business. Hiring more people now focuses on making sure that they will be able to work with this stuff, because it's what the business is already using after all. And to be attractive, new engineers are trained to use those frameworks.. and now the cycle just repeats itself.
Moral of the story: it's a mix of race to the bottom, commodification, shortage and standardisation. That last one is important, because using standards is preferred in a risk-averse environment, and most businesses are just that.
If you have a job posting describing "work on our website" you get all sorts of random candidates that have no real place in your company, but when you say "developer with framework XYZ experience" you can skip everything, judge their framework experience and move on (or so it is thought to work).
This means that to be attractive you need to use it as a developer, but inside a company this stuff has to be used to make sure that the current developer(s) can be replaced easily (not how it realistically works beyond CRUD apps), and expanding the workforce is easier because attracting developers with technology keywords is easier than business job descriptions.
End result: using technology not because it's the best fit, but because it is off-the-shelf. Essentially a modern version of using Oracle and Windows NT 3.51 back in the day, you're using it because it's an easy off-the-shelf keyword, not because it actually has anything to do with what you are technically trying to get done. Enterprise IT and HR shaking hands all over again.
Some more expansion on this topic (story time):
Generally when you are large enough as a team or business unit to have lots of collaboration, you need some way to exchange information and ideas, and work together using similar-ish principles. That can mean various things, from all using the same OS and IDE to having up-to-date API documentation and integration tests. This means that if you have to transport some idea, project or application across people (and teams) you need a method and format to do so.
This nearly automatically also means you can't do solo yolo all day and you need to have rules and packaging standards so you don't end up figuring out what Pete did on his machine so it works on Jack's machine but not Jane's. Chroot-in-a-tarball works, virtual machine images, maybe autotools with a bunch of setup scripts. But most of the older methods all have the same problem: your systems all need to be rather similar to work with it, and you need to have systems knowledge to make use of it.
With the expansion of the workforce, that becomes more and more scarce, to the point where most Junior and Medior engineers have no clue how their computer actually works, how to manage an operating system, how how to understand the influence of externalities (be it them influencing other teams, or the other way around). People with less experience are easier to 'create' (yes, I know, we're not actuallt manufacturing people on-demand for work), as it requires less affinity, less time, and less education resources. So we can get more of them, and as long as they are put in an environment where they don't need to be as capable as 'full fat' engineers, the business can keep running.
So, what do we do? We package and standardise. First with environments, we have virtual machines, images, packer, and can see those local virtual machines with ansible and saltstack and if it works there, the project will work on other workstations and servers just the same, since it was all provisioned the same. No more guessing. But this is a really heavy method of doing this, and the greybeards will complain that just using some file protocol to directly edit something or a simple jail or chroot was much lighter (faster, less resources etc). At the same time, from a security perspective, sandboxing stuff gets more traction, and with namespaces and control groups we also get docker (pre-OCI), which gets a bunch of people excited because you can now make transportable environments that are essentially jails, but without having to know anything about jails or bsd and it works on Linux!
Parallel to this, we get various tools for dependency management, mainly derived from the same concepts that drive dpkg/apt and rpm/yum, because it worked so well for system packages, why not use it for nearly every language under the sun? This gets into DLL-hell rather quickly, contaminates the operating system by installing packages needed for your projects globally for the entire system, so now whenever you switch to a different projects you basically have to reinstall. Boo! But luckily someone heard of this fancy new container thing and you can just do all of that stuff in there!
Now we can simply create those project environments as container images (got some real traction once OCI was a thing), so an engineer working on some new feature has even less of a requirement regarding their skillset. Just hire someone who can run docker containers and work on framework XYZ! Tons of those available. So instead of having teams with 5 highly skilled workers with deep knowledge and a high cost, you just get 3 Juniors for bulk work, 1 medior for checking their work and 1 senior that's in that classic "know as much as possible" bucket do mitigate any complex issues that may arise.
Fast forward to today, the new greybeards are the senior engineers of a decade ago, less and less new people with deep knowledge have been added to the workforce, but large masses of either highly specialised or shallowly trained people are being hired. They are not unskilled, but just skilled for a specific piece of the business, because that works for other departments so why not IT? And there is a shortage after all...
The now rare and in high-demand deep knowledge engineers are tired of getting the same questions for the same architectural problems and same business complexities, so they come up with software frameworks that they can apply to the problem. It doesn't matter if the other team members have a varying gradient of understanding of the details of the solution, as long as the tickets get closed and added value is perceived by non-technical people. Those same people may see those framework names, and recognise them in various important places around the business. Hiring more people now focuses on making sure that they will be able to work with this stuff, because it's what the business is already using after all. And to be attractive, new engineers are trained to use those frameworks.. and now the cycle just repeats itself.
Moral of the story: it's a mix of race to the bottom, commodification, shortage and standardisation. That last one is important, because using standards is preferred in a risk-averse environment, and most businesses are just that.