I’m not sure why this is a top post. The definitions of controller and operator are completely wrong. The example code is for creating a custom api server which is only done in the most advanced of advanced use cases. The implementation of the apiserver is too naive to demonstrate they have any understanding of the complexity that supporting watch will cause.
The article has a description of what an operator is wrong. The definition of an operator originally was...
> An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts but includes domain or application-specific knowledge to automate common tasks.
This is the original definition of an operator [1]. People no use them for stateless things and domain specific work has taken off.
You can look at the Kubernetes docs [2] to see refinements on it...
> Kubernetes' operator pattern concept lets you extend the cluster's behaviour without modifying the code of Kubernetes itself by linking controllers to one or more custom resources. Operators are clients of the Kubernetes API that act as controllers for a Custom Resource.
They watch for their CR changing and then act on it - usually by making requests to the API server to create/update/delete objects (e.g. a operator for a database might make Pod creation requests when it spots the FooDatabase CR being created)
In essence, a controller is like a general-purpose manager for built-in K8s resources, while an operator is a specialized manager tailored for specific applications and tasks. Operators are built on top of the controller pattern, so you can think of them as the next level up in the K8s automation!
but i will agree on that i have seen people develop controllers with the operator sdk where mostly the Kubernetes client libraries would have been more than enough.
Conceptually, an Operator is intended to embody the knowledge and processes of a human operator for a given functionality. Extending the Kubernetes API might be an implementation detail but it’s entirely optional to extend it. Most Operators do extend it in practice, however.
A better way to write an operator these days is to use kubebuilder [1].
My complaint is that I have seen orgs write operators for random stuff, often reinventing the wheel. Lot of operators in orgs are result of resume driven development. Having said that it often comes handy for complex orchestration.
Your app becore deployment needs a complex monitoring stack making use of many external PaaS products.
Normally you would have a CI/CD pipeline setup that shit before deployment.
But thats super brittle and against GitOps. So you write custom CRDs you can store in repo that abstract away the complexity of the setup but still provide devs with srlf-service approaches to configure their app monitoring for them.
If you start thinking about self-service and gitops - operators become very useful for many things.
An operator operates something, e.g. it actively makes changes. If you want to deploy an application, a Helm Chart is the correct way. It will allow you to have deterministic deployment, that you can duplicate multiple times in your cluster, and you can dry-run it and see the generated manifests.
An operator is needed when you can't just deploy and forget about it. An example is the Prometheus operator, which will track annotations created by users to configure the scraping configuration of your Prometheus instances. Another example is cert-manager, which gets certificates into secrets based on Certificate and Ingress objects, renews them automatically before expiry, and does that by creating ingresses picked up by your ingress controller.
The advantage of an operator is that it will react to stuff happening in the cluster. The drawback is that it reacts to stuff happening, potentially doing unexpected things because changes happen at any time and you can't dry-run them. Another drawback is that they are usually global, so you can't run multiple versions at the same time for different namespaces (mainly because custom resource definitions are global).
Unfortunately many people think packaging an application = creating an operator, and that operator does nothing a chart couldn't do.
The CockRoach DB example in the article is a perfect example of an unnecessary CRD. Acquiring certificates within an Kubernetes cluster is a common requirement for lots of applications and there are lots of solutions out there. Is it really necessary to spend time writing your own operator? Now you have a second helm chart and an operator to maintain. Now you have to explain to people which chart to use. You could get rid of the non-operator chart but now I have operators within the cluster acquiring certificates in 5 or 6 different ways. Do I have to configure the credentials for 6 operators so they can make Route53 DNS challenge records?
Edit: maybe we could shift left and ask the app developers to add certificate acquisition directly into the app source.
That’s true. The actual process of granting a pod access to edit route53 TXT records is pretty easy.
The problem is duplication of functionality within the cluster and increased complexity in the configuration of deployments. I want to configure a certificate acquisition process once, and plug those certificates into pods using a generic process based on annotations. I don’t want to configure it 6 times because each operator does things slightly differently. Each CRD introduces new complexity to the cluster. More opportunities for things to break. More stuff to read and learn about.
I think CRDs should be used to extend the platform functionality, not as a tool to simplify deployments. It feels a little like reworking the plumbing in your house to make the sink easier to install instead of using a wrench.
Or is can be actively harmful when they don't do any error checking whatsoever, causing it to be less accurate that `helm template` would be. Related, it's also one more thing to monitor because it can decide to start vomiting errors for whatever random reason
Neither of those cases really need an operator -- Prometheus and cert-manager both have code that watches for changes on ingresses/services/custom resources and reacts to changes (using permissions granted via RBAC). I've used both without an operator and still use Prometheus without one.
Everyone else commenting is missing the point, because as you've said, Prometheus can discover targets via k8s endpoints/pods dynamically already, and you can run a sidecar to reload the config file.
The main point of Prometheus operator is to federate access to Prometheus configuration so teams can manage configuration that the built-in Kubernetes service discovery doesn't let you control from an application side. Things like scrape interval, recording rules and alerts, etc. ServiceMonitors, PrometheusRule CRDs basically let app owners ship these aspects of monitoring with their applications, instead of having to have a hand-off between the service owners and the SRE teams managing the monitoring infrastructure.
Pronetheus does not change its config if not asked to do it… and in most cases scrape config is stored in a configmap that does not change by itself magically.
So yes -> you need an operator to dynamically change config of prometheus. If not -> all your changes wont matter because they will be not reflected in prom.
There are several Prometheus helm charts out there. The one I use runs a sidecar container that watches for config changes and tells Prometheus to reload the config. So you don't need the operator - there are other approaches.
Wdym the configmap doesn't change by itself? I don't want my configurations changing "magically" without my say so
The two features I need (and can get without the provider) are:
1. If I (manually, explicitly) change the configmap, prometheus can pick it up without being restarted. This is provided by the side-car container.
2. If new pods come online they are automatically detected and scraped by Prometheus. As long as they have the annotations saying: a) this pod should be scraped, b) scrape on this port, and c) scrape at this URL - then they will be scraped.
I dunno what to tell ya, I see pods come and go from Prometheus all the time, including those from newly added Service objects with Prometheus annotations. I'm using kubernetes_sd_configs.
You don't NEED an operator, but there are benefits.
If all of your pods have the same scrape settings, there's not much benefit to the operator. But if each pod or set of pods needs custom scrape settings - say different scrape intervals or custom tag re-writes, the operator will let you define each set of scrape configs separately as kubernetes resources. You can store the definitions alongside the resources they scrape instead of having a single large complex config in your prometheus deployement. This would be especially beneficial if you have multiple projects owned by multiple teams, all scraped by the same Prometheus.
Granted I also don't use the operator, but I've looked into it. With complex enough deployments it would simplify things.
I worked on an operator that manages Kafka in K8s. If you want to upgrade the brokers in a Kafka cluster, you generally do a rolling upgrade to ensure availability.
The operator will do this for you, you just update the version of the broker in the CR spec, it notices, and then applies the change.
Likewise, some configuration options can be applied at runtime, some need the broker to be restarted to be applied, the operator knows which are which, and will again manage the process of a rolling restart if needed to apply the change.
You can also define topics and users as custom resources, so have a nice Gitops approach to declaring resources.
i know what you mean, but i am doing this for more than 20 years now. From bare-metal over openstack to serverless, i have pretty much provisioned all of them.
Kubernetes is more like a way of doing things than a technology. Basically APIs all the way down. and thus the operators and controllers do deserve love.
Im not saying that you need an operator to change the dipers of a baby, but as far as a stack goes, k8s is the best i have ever worked with.
A good example from my perspective is when you are delivering an application as 3rd party vendor and you wish to automate lot of operational stuff like backup, scaling based on events, automating stuff based on cluster events. It starts becoming very valuable. I am sure there are many more use cases for.
I would not write an operator to do any of these things. To me an "operator" strongly implies the existence of a CRD and the need to manage it. So for autoscaling, HPA/VPA are built into k8s. Backups should be an application-level feature; when the "take a backup" RPC or time arrives, take a backup and dump it in configured object storage. Automating stuff based on cluster events also doesn't require an operator; call client.V1().Whatever().Watch and do what you need to do.
The only moderately justifiable operator I've ever seen is cert-manager. Even then, one wonders what it would be like if it just updated a Secret every 3 months based on a hard-coded config passed to it, and skipped the CRDs.
Operators make sense when you need to automatically modify resources in response to changes in the cluster's state.
An example that has come up for me is an operator for a Kafka Schema Registry. This is a service that needs some credentials in a somewhat obscure format so it can communicate very directly with a Kafka broker. If the broker's certificates (or CA) are modified, then the Schema Registry needs to have new credentials generated, and needs to be restarted. But the registry shouldn't (obviously) have direct access to the broker's certificates. Instead, there's a more-privileged subsystem which orchestrates that dance; that's the operator.
kubernetes itself is a collection of controllers/operators. It takes manifests like pods and uses that information to create the workload in your container runtime on a node with the resources it needs.
Also it's important for people reading this article - an apiservice (which this article talks about) is very rarely something that should be done. An operator is more appropriate for nearly all cases except for when you truly need your state stored outside of the internal Kubernetes etcd datastore.
Since Go got generics, working with the Kubernetes API could become far more ergonomic. It's been pulling teeth until now. I'm eager to see how the upstream APIs change over time.
In the mean time, one of the creators of the Operator Framework[0] built a bunch of useful patterns using generics that we used to build the SpiceDB Operator[1] called controller-idioms[2].
Does anyone know of other efforts to improve the status quo?
This was part of why I chose Python with the last operator I wrote. Granted, it was a slow process that wasn't time-sensitive (and couldn't be paralleled), so performance wasn't a concern. Regardless, I avoided Go because of previous experiences with how hard a lack of generics made working with such things.
I'm hoping things are improving. Maybe I'll consider it for the next one I write.
I didn't really enjoy my experience with the few operators I've worked with, mainly because they require the maintainer to build in some sort of access to basic kubernetes functionality. I see the benefit of operators, but I hated that in order to do something as simple as define memory/CPU limits to certain containers I would need to open a PR to the repo and wait weeks, sometimes months, for a new release.
It's frustrating to be a kubernetes admin but not have access to basic configuration options because the maintainers of even some very high-profile operators (looking at you, AWX) neglected to build in access to basic functionality.
In the latest release of the spicedb-operator[0], I added a feature that allows users to specify arbitrary patches over operator-managed resources directly in the API (examples in the link).
There are some other projects like Kyverno and Gatekeeper that try to do this generically with mutating webhooks, but embedding a `patches` API into the operator itself gives the operator a chance to ensure the changes are within some reasonable guardrails.
I'm not saying they shouldn't customize their PodSpec, I'm saying if they do, your controller is exactly the same as the built-in Deployment controller, running user-provided pods. Why build a copy of it?
The SpiceDB operator looks like a prime example of something that should have been a Helm Chart. Migrations can be run in the containers.
Operators are just the non-containerized daemons of the Kubernetes OS. We did all this work to run everything in neatly encapsulated containers, and then everyone wants to run stuff globally on the whole cluster. What's the point? Do we just containerize clusters and start over?
I get the sentiment. We held off on building an operator until we felt there was actually value in doing so (for the most part, Deployments cover the operational needs pretty well).
Migrations can be run in containers (and they are, even with the operator), but it's actually a lot of work to run them at the right time, only once, with the right flags, in the right order, waiting for SpiceDB to reach a specific spot in phased migrations, etc.
Moving from v1.13.0 to v1.14.0 of SpiceDB requires a multi-phase migration to avoid downtime[0], as could any phased migration for any stateful workload. The operator will walk you through them correctly, without intervention. Users who aren't running on Kubernetes or aren't using the operator often have problems running these steps correctly.
The value is in this automation, but also in the API interface itself. RDS is just some automation and an API on top of EC2, and I think RDS has value over running postgres on EC2 myself directly.
As for helm charts, this is just my opinion, but I don't think they're a good way to distribute software to end users. The interface for a helm chart becomes polluted over time in the same way that most operator APIs become polluted over time, as more and more configuration is pulled up to the top. I think helm is better suited to managing configuration you write yourself to deploy on your own clusters (I realize I'm in the minority here).
I'm not sure what you're on about. Operators don't need to run in cluster at all. And even then, they can absolutely run as containers. And as far as permissions go, that's up to you. They're just regular service accounts.
Custom resource definitions are global to the cluster, and operators are usually written to watch those resources in the whole cluster.
You run the operator itself in a pod, but you can only have one of them running at a time, hence they are global. You can run as many Alpine/Ubuntu/... containers as you want, and as many ElasticSearch or PostgreSQL as you want, in isolation, that's the beauty of containers. But you can only run one SpiceDB operator or one ElasticSearch operator, on the whole cluster. We're back to square one.
I don't think it's quite as bad as you're suggesting. Operators are control-plane, your other examples are data-plane. You can also only run one Deployment controller in a kube cluster, but many Deployments. And you can run many controllers for a single CRD, see the Gateway APIs for example, but it's not as common and is more work.
That said, I do think the direction that Kube has gone with CRDs, by leaning into making them more like built-in apis instead of allowing them to be differentiated as an external extension point has limited what you can do with them.
When ThirdPartyResrouces just carved out a group and a name in the API it was simple to have multiple copies and multiple versions running together in a cluster. But that has become more difficult with time, especially with ConversionWebhooks - a feature that forces all controllers for an API to agree not just on a version, but on the specific ways versions convert into each other.
> I would need to open a PR to the repo and wait weeks, sometimes months, for a new release.
Just curious, is this a limitation of the Operators framework, or that of your system's implementation? My knee-jerk reaction is that any implementation should absolutely not require opening ticket. After all, Amazon's API mandate happened 20 years ago, and Netflix followed suit to achieve phenomenal productivity for their engineers. I have a hard time imagining why any engineer would think that gatekeeping configuration with PR is a good idea(a UI with proper automation and approval process that hides generated PR for specific use cases is a different matter)
Not a kubernetes expert, but my understanding is that that operators are regular programs that run in a kubernetes container and interact with the kubernetes API to launch/manage other containers and custom kubernetes resources.
An operator (or its custom resource) can be configured by Kubernetes YAML/API and its upto the creator of the operator to specify the kind of configuration. If the operator creator did not specify options to set cpu/memory limits on the pods managed by the operator, then you can't do anything. You have to add that feature into the operator and then make a pull request and wait for it to be upstreamed.
Or fork it instead. Same thing for helm charts (except forking and patching them is easier than forking an operator).
You have a problem: orchestrating some thing in kube, so you write some custom operator logic running alongside your main product; but now you have two problems to worry about.
I've seen just as much if not more issues with debugging the operator logic itself as with the main pods/deployments it was trying to manage.
So just from a practical point of view, I think it should be a last resort after everything else fails (helm charts, etc).
Oh i love operators they usually tie the entire cluster together and lead to amazing things!
Think of Kubernetes as an advanced API server that can be extended endlessly and operators are the way to do it.
There really is no magic, is all there and with go the images are usually what? like 10 mb?
It's essential to have a solid understanding of Kubernetes architecture, concepts such as custom resources and controllers, and the tools and APIs available for working with Operators.
Dont use rust though, use and sdk like the operator sdk or kubebuilder. Its native to k8s and you will have a much easier time too.
It should be helpful if you are new to the Operator concept.
Operators are generally useful for handling domain-specific actions - for example, performing database backups, installing plugins on Moodle/Wordpress, etc. If you are looking for application deployment then a Helm chart should be sufficient.
Using Rust for that is a bad idea, just use the official and native SDKs ( in Go ). Rust does not have any equivalent to https://sdk.operatorframework.io/
I've written (well, participated in development of) two Kubernetes operators, and support about a dozen of them (in our own deployment of Kubernetes): Jupyter, PostgreSQL, a bunch of Prometheus operators and a handful of proprietary ones.
In my years of working with Kubernetes I cannot shake the feeling that it's, basically, an MLM. It carefully obscures it's functionality by hiding behind opaque definitions. It doesn't really work, when push comes to shove. And, most importantly, it survives in a parasitic kind of way: by piggybacking on those who develop all kinds of extensions, be it operators, custom networking or storage plugins, authentication and so on.
My problem is I cannot find who stands at the top of the pyramid. There's Cloudnative Foundation, but all it does is selling certifications nobody really needs... so, that cannot possibly be it. No big name doesn't really benefit from this in an obvious way...
So... anyways, when I hear people argue about how to implement this or another extension of Kubernetes, it rings the same as when people argue about styles of agile, or code readability etc. nonsense. There isn't a good way. There is not acceptance criteria. The whole system is flawed to no end.
I find myself conflicted between two approaches at work:
1. Write a provider/extension/whatever for a tool like Terraform or Pulumi. I live in a world where the infrastructure doesn't move underneath my feet. I am the source of truth. I feel like I only need to reconcile changes when I make changes to my IaC repositories.
2. I could write something that exists in a control plane, like Kubernetes operators or Crossplane. I live in a world where I look at the world, find the delta between current state and desired state, then try to reconcile. This is an endless loop.
I feel like these are different approaches with the same goal. Why should I decide either way beyond tossing a coin?
Some use cases:
- an internal enterprise DNS system which is not standards-compliant with the world at large
- an internal certificate authority and certificate issuing system.
Terraform doesn't run a control loop. If you've never been in a situation where you've needed a control loop to reconcile the deployment state, then you've probably never needed an operator. In most cases, the native controllers (and resources) from Kubernetes are enough, but sometimes, an application requires additional hand-holding. I find this to be especially true for legacy applications.
