I find all the complexity quite unnecessary. I think the "package managers" attempt to do too much, handling some sort of use case that doesn't affect me. My highest success rate with randomly deploying someone else's software comes from software that just provides a bunch of API objects in YAML form that you just apply to the cluster. My second highest success rate comes from just writing my own manifest for some random docker image. Finally, operators tend to do pretty well if the development team is sane.
At this point, I think the fundamental problems are:
1) People desire to give you a "works out of the box" experience, so they write a packaging system that can install everything. The app depends on Postgres? Fuck it, we'll install Postgres for you! This is where things start to go wrong because self-hosting your own replicated relational database instance is far from trivial, and it requires you to dial in a lot of settings. And, of course, it requires even more settings to say "no no, I already have an instance, here is the secret that contains the credentials and here is its address."
2) Installing software appears to require answering questions that nobody knows the answers to. How much CPU and memory do I give your app? "I dunno we didn't load test it just give it 16 CPUs and 32G of RAM, if it needs more than that check your monitoring and give it more." "I only have 2 CPUs and 4G of RAM per node." "Oh, well, maybe that's enough or maybe it isn't. It won't blow up until you give it a lot of users though, so you will get to load test it for us while your users can't do any work. Report back and let us know how it goes!"
I also noticed that when security people get at the project, it tends to become unusable. I used to be a big fan of Kustomize for manifest generation. Someone decided to build it into kubectl by default, and that it should support pulling config from random sites on the Internet. So now if you use it locally, you can't refer to resources that are in ../something, because what if a remote manifest specified ../../../../../etc/shadow as the config source? Big disaster! So now it doesn't work. (They also replaced what I thought was the best documentation in the world, a kustomization.yaml file that simply used every available setting, with comments, with a totally unusable mass of markdown files that don't tell you how to use anything.)
Obviously security is a problem, but they should have said "just git clone the manifest yourself and review it" instead of "you can't use ../ on your local manifests that are entirely written by your own company and exist all inside the same git repository that you fully control". But they didn't, and now it sucks to use.
At this point, I think the fundamental problems are:
1) People desire to give you a "works out of the box" experience, so they write a packaging system that can install everything. The app depends on Postgres? Fuck it, we'll install Postgres for you! This is where things start to go wrong because self-hosting your own replicated relational database instance is far from trivial, and it requires you to dial in a lot of settings. And, of course, it requires even more settings to say "no no, I already have an instance, here is the secret that contains the credentials and here is its address."
2) Installing software appears to require answering questions that nobody knows the answers to. How much CPU and memory do I give your app? "I dunno we didn't load test it just give it 16 CPUs and 32G of RAM, if it needs more than that check your monitoring and give it more." "I only have 2 CPUs and 4G of RAM per node." "Oh, well, maybe that's enough or maybe it isn't. It won't blow up until you give it a lot of users though, so you will get to load test it for us while your users can't do any work. Report back and let us know how it goes!"
I also noticed that when security people get at the project, it tends to become unusable. I used to be a big fan of Kustomize for manifest generation. Someone decided to build it into kubectl by default, and that it should support pulling config from random sites on the Internet. So now if you use it locally, you can't refer to resources that are in ../something, because what if a remote manifest specified ../../../../../etc/shadow as the config source? Big disaster! So now it doesn't work. (They also replaced what I thought was the best documentation in the world, a kustomization.yaml file that simply used every available setting, with comments, with a totally unusable mass of markdown files that don't tell you how to use anything.)
Obviously security is a problem, but they should have said "just git clone the manifest yourself and review it" instead of "you can't use ../ on your local manifests that are entirely written by your own company and exist all inside the same git repository that you fully control". But they didn't, and now it sucks to use.