Not gonna lie. I designed a system that looks exactly like this. However, the plugin system was an operator instead of annotating a service which is kinda cleaner but it still felt dirty. A better approach is having a service registry to track the versions, values and clusters where this has been pushed but that means you needed a different API or extend the k8s API and deal with the whole auth/authz system.
I think the biggest hurdle to implement a solution like this is your application MUST be a 12 factor app which sadly is not the case from what I've seen. Many devs hardcode values when shipping their code and containers which make it really difficult to add nodes to a DAG. Had this change, SDLC testing would be waaay easier.
In the same space, labeling of all services is a thing. As you are annotating it can be tracked but if you had different gateways or destination routes or other things that need to be tweaked at the mesh level, it could be daunting to track it. In my case, I ensure the whole mesh representation of a version could be tracked by a label so if you had a app=vote1,version=release-4,feature=shiny you could track the shiny components across the whole cluster.
Another hurdle is you are tied to a service mesh implementation. Istio is ok but it can be a beast. It also constraints your org to implement other networking alternatives which is something we wanted to explore.
Man this brilliant and goes straight to test it with my development workflow. Namespace isolation was always PITA,but full isolation would require a dedicated clusters so having a best of both solution really changes the game for lean and cost effective continues delivery.
As for value prop, maybe emphasize multi tenancy a bit more as this has the most cost saving potential at scale.
For sure, probably test it over the weekend. I've dealt with k8s virtualization before and as a result, my current system uses contextual auto configuration for all it's services which makes it very easy to lift it from one kubernetes context to another, or even a docker only environment as it's often the case with CI.
However my primary challenge is chronic cluster under utilization after having rewriten the bulk of the system in Rust. Therefore virtualizating the cluster makes the most sense. I think Google Cloud conducted a study that found that the bulk, like 75%, of their K8s customer over provision by a wide margin. So there is definitely a market for cluster virtualization and even more so for multi tenancy.
Reading the docs, I consistently get blank page with an error after less than 30s: "Application error: a client-side exception" it feels strange that a text document needs to be an Application. I feel so old.
How is this different from vcluster? In fact vcluster has no dependency on Istio which I think is a huge detriment to kardinal.
We use cilium so the istio requirement becomes a non starter for us.
Regarding how Kardinal compares with vcluster, both aim to provide multitenancy within a single Kubernetes cluster, but they approach it differently.
vcluster is designed to offer a full Kubernetes experience to each tenant, including their own API server, control plane, and other Kubernetes-level components. This makes vcluster an excellent choice if you need strong isolation or want to test Kubernetes-specific features like custom CRDs or operators.
Kardinal, on the other hand, focuses on application-level multitenancy rather than providing the full Kubernetes stack to each tenant. This makes Kardinal lighter-weight and more efficient if you're primarily concerned with deploying and testing application-level changes.
Tilt works as an in-place hot-reload tool for containers. However, because it performs in-place updates, it doesn't support running multiple parallel and isolated versions of your cluster side-by-side.
That said, the timing of your question is perfect since we are about to push a Tilt integration that allows you to create an isolation flow and use Tilt to hot-reload and update that flow in-place! :)
You're completely right, ofrzeta! Kardinal does manage Istio for you, so you don't need to learn or interact with Istio at all. In fact, Kardinal acts as a transpiler, taking Kubernetes manifests with simple Kardinal annotations and generating and applying the Kubernetes and Istio resources for you.
This looks interesting, but is there a reason the kardinal-cli deploys kardinal-manager into the default namespace rather than taking a --namespace argument (for example)? Personally I never run anything in the default namespace but I am probably an outlier.
Congratulations on shipping! I like how opinionated Kardinal is, which means it should work nicely for anyone who shares the same kind of infras vision as you.
It's also such an interesting moment for you folks to show up on HN, I just shipped the first preview version of my Kubernetes operator(https://github.com/pier-oliviert/sequencer) that manages ephemeral environments. I can see some things that are similar with both our options as well as some things that are quite different.
Maybe if I had one question is: What made you go for Istio as the main network mesh?
Nice work on Sequencer! I really liked the Workspace CRD.
Nothing too specific on choosing Istio, but it seemed like a popular and battle-tested option to start the implementation with. Kardinal is a fairly young project (about 2 months old), and we expect it to adopt more "service mesh backends" in the future.
Looks interesting. I currently deploy dev instances as a single pod with everything in it, basically exactly the same as what devs get if they run docker compose locally. It works but pretty wasteful for many cases that don't need their own db etc. Is this supposed to make that less wasteful? Is it any different in the "full app"/end to end mode?
Yes, Kardinal is designed to make development environments more efficient and less wasteful compared to your current setup, where each dev instance is a single pod with everything in it. In Kardinal, we calculate reusability at the application level. You only deploy the services you're actively working on, which means you won't need to spin up unnecessary components, like a separate database, if it's not needed for your dev/test case. This will improve both cost and speed.
That sounds amazing. Maybe a before/after picture would be nice for the front page, showing the "one whole pod per instance" setup vs the "only the bits you need" setup.
There is an alternative and complementary approach worth considering - instead of duplicating infrastructure in a smaller scale, you can simplify it to the simplest possible requirements needed to make your service run. Consider an old shared hosting - a single server could run 100s of wordpress websites just as well.
unfortunately, this tool assumes full coverage with k8. but let's say you have an external pub/sub solution and your services are decoupled. that's a pretty low bar.
sidenote: I noticed Istio (envoy actually) has some weird non-deterministic behavior when you hit pod resource limits (504 bad gateway, 0DC)
Not sure if I'm addressing your point correctly, but Kardinal does have the concept of external services and provides a plugin system that allows you to manage stateful external services. Here is an example [1] of a plugin to manage Neon [2] (a serverless Postgres DB with support for branches). You can find more information about plugins here: https://kardinal.dev/docs/concepts/plugins
namespace-based deploys (or telepresence too) is equally "lightweight" to Kardinal. it's just that (at least in our phrasing), those don't quite constitute a separate "environment" as state is shared between any developers working at the same time
Kardinal matches those approaches in terms of light-weightedness, but offers state isolation guarantees too (like isolation for your dbs, queues, caches, managed services, etc). so in comparison to "ephemeral environment" approaches that give state isolation, we do believe we're doing this in the most lightweight way possible by implementing that isolation at the layer of the network request rather than by duplicating deployed resources
wow, kardinal sounds cool for consolidating pre-production clusters! let's stop duplicating and start simplifying our kubernetes environments. who else is all in for the hyper-lightweight multitenancy framework trend? let's discuss!
I think the biggest hurdle to implement a solution like this is your application MUST be a 12 factor app which sadly is not the case from what I've seen. Many devs hardcode values when shipping their code and containers which make it really difficult to add nodes to a DAG. Had this change, SDLC testing would be waaay easier.
In the same space, labeling of all services is a thing. As you are annotating it can be tracked but if you had different gateways or destination routes or other things that need to be tweaked at the mesh level, it could be daunting to track it. In my case, I ensure the whole mesh representation of a version could be tracked by a label so if you had a app=vote1,version=release-4,feature=shiny you could track the shiny components across the whole cluster.
Another hurdle is you are tied to a service mesh implementation. Istio is ok but it can be a beast. It also constraints your org to implement other networking alternatives which is something we wanted to explore.
I do like the project uses Nix =).