Right, I'm talking about shared-kernel multitenancy. Shared-kernel multitenancy isn't just about reducing the OS overhead from host + one or more VMs (or sandboxes) to just a single host, it's also about not having to continually start and stop virtual machines/sandboxes, which introduces its own resource overhead as well as a latency hit (which essentially always coincides with resource pressure from increased usage, since that's why you're scaling up) every time it's done. Also, even VMs and sandboxes don't really protect against DoS/resource fairness/noisy neighbor problems that well in many cases.
Why does this matter? Incurring kernel/sandbox boot overhead on cold start/scaling makes it so that services have to over-provision resources to account for potential future resource needs. This wastes a lot of compute. I also think it's incredibly wasteful for companies to manage their own K8s cluster (if K8s supported multitenancy you'd probably want only one instance per-datacenter, and move whatever per-cluster settings people depend on to per-service. This is also much closer to "how Google does things" and why I think Kubernetes sucks to use compared to Borg), again because of all the stranded compute, and also because of the complexity of setting it up and managing it - but without shared-kernel multitenancy, multi-tenant K8s has to employ a lot of complicated workarounds in userspace. Or you can use a serverless product, ie pay for someone else's implementation of those workarounds, and still suffer some of the resource/latency overhead from lack of shared kernel multitenancy.
This is one of the problems I want to address with my company BTW, but it would take years if not decades to get there, which is why I'm starting with something else.
Why does this matter? Incurring kernel/sandbox boot overhead on cold start/scaling makes it so that services have to over-provision resources to account for potential future resource needs. This wastes a lot of compute. I also think it's incredibly wasteful for companies to manage their own K8s cluster (if K8s supported multitenancy you'd probably want only one instance per-datacenter, and move whatever per-cluster settings people depend on to per-service. This is also much closer to "how Google does things" and why I think Kubernetes sucks to use compared to Borg), again because of all the stranded compute, and also because of the complexity of setting it up and managing it - but without shared-kernel multitenancy, multi-tenant K8s has to employ a lot of complicated workarounds in userspace. Or you can use a serverless product, ie pay for someone else's implementation of those workarounds, and still suffer some of the resource/latency overhead from lack of shared kernel multitenancy.
This is one of the problems I want to address with my company BTW, but it would take years if not decades to get there, which is why I'm starting with something else.