During many years of operating several-thousands of nodes production clusters on Kubernetes, I've never seen any of these observability tools that query kube-apiserver work at that scale. Even the popular tools like k9s make super expensive queries like listing all pods in the cluster that if you don't have enough load protections, can tip your Kubernetes apiserver over and cause an incident. If you're serious about these querying capabilities, I highly recommend building your own data sources (e.g. watch objects with a controller and dump the data in a sql db) and stop hitting apiserver for these things. You'll be better off in the long run.
There is a funny parallel I see with Kubernetes that I also saw a lot with Linux in the early years. There are thousands of packages and tools you can install on Linux (think phpmyadmin for example) and new users sometimes go wild installing every single package they read about.
After a while, the more mature Linux engineers start going the other way. Ripping out as much as possible. Stripping down to the leanest build they can, for performance but also to reduce attack surface and overall complexity.
Very similar dynamic with k8s. Early days are often about scooping up every CNCF project like you're on a shopping spree. Eventually people get to shipping slim clusters running and 30mb containers with alpine or nix. Using it essentially as open source clustering for Linux.
What's surprising to me is that there's no way to listen to any object type. You have to know the "kind" beforehand, because the watch API requires it. To watch all objects in the system, you have to start a separate watch request for every type. This may in turn be expensive.
If you have direct access to Etcd (which may not be possible in a managed cloud version of Kubernetes?), putting a watch on / might scale better.
(As an aside, with the Go client API you have to jump through some hoops to even deserialize objects whose kinds' schemas are not already registered. You have to use the special "unstructured" deserializer. The Go SDK often has to deal with unknown types, e.g. for diffing, and all of the serializer/codec/conversion layers in the SDK seem incredibly overengineered for something that could have just assumed a simple nested map structure and then layered validation and parsing on top; the smell of Java programmers is pretty strong.)
The watch API has horrible user experience in all platforms. One must send a GET and keep the pipe open, waiting for a stream of responses. If the connection is lost, changes might be lost. If one misses a resource version change, then either the reconnection will fail, or a stale resource will be monitored.
The Java client does this with blocking, resulting in a large number of threads.
I truly like Kubernetes, and I think most detractors' complaints around complexity simply don't want to learn it. But the K8s API, especially the Watch API, needs some rigorous standards.
how are Kubernetes apiservers suffering this much from this kind of query? Surely even in huge systems the amount of data that would need to be traversed is super small, right?
Is this a question of Kubernetes just sticking everything into "standard" datastructures instead of using a database?
My knowledge is out of date now, but the main issues IMO are/were:
- No concept of apiserver rate limiting, by design. I see there is now an APF thingy, but still no basic API / edge rate limiting.
- etcd has bad scalability. It's a very basic, highly consistent kv store that has tiny limits (8GB limit in latest docs, with a default of 2GB). It had large performance issues throughout its life when I was using k8s, I still don't know if it's much better.
Long ago I wanted to re-implement at least part of kubectl in Python. After all, Kubernetes has documented API... what I quickly discovered was that kubectl commands don't map to Kubernetes API. Almost at all. A lot of these commands will require multiple queries going back and forth to accomplish what the command does. I quickly abandoned the project... so, maybe I've overlooked something, but, again, my impression was that instead of having generic API with queries that can be executed server-side to retrieve necessary information, Kubernetes API server offers very specialized disjoint set of commands that can only retrieve one small piece of interesting info at a time.
This, obviously, isn't a scalable approach, but there's no "wrapper" you could write in order to mitigate the problem. The API itself is the problem.
Pretty sure the apiserver just queries the etcd database (and maybe caches some things, not sure) but i guess it could be the apiserver itself that can't handle the data :P
Kubernetes only lets you query resources by object type and that's only a prefix range scan on etcd database. There are no indexes whatsoever in the exhaustive LIST queries, and kube-apiserver handles serialization of the objects back and forth between multiple wire types. Over the years there has been a lot of optimizations, but you don't wanna list all pods in a 5000 node high density cluster every time you spin up client-side tools like this.
In my experience, they don't, you can just run more of them and you can stick them behind a load-balancer (regular HTTP reverse proxy). You can scale both etcd and apiserver pretty easily. Of course you have less control in cloud environments, I have less experience with that.
This is the approach we took while building our Internal Developer Platform: watches (via client-go informers with client-side caching) to sync data into a Postgres database as JSONB. Changes are tracked using JSON patches and Kubernetes events. To avoid a watch on every resource kind, we handle this by performing incremental object fetches for the objects involved in watched events.
Getting this to perform well required several optimizations at both the Go and Postgres levels. On the Go side, we use prioritized work queues, event de-duplication, and even switched to Rust for efficient JSON diffs. For Postgres, we leverage materialized views and trigger-based optimistic locking
That's how https://github.com/KusionStack/karpor did it. It has a resource-syncer component to synchronize resources in real time to Elasticsearch, and then allows users to search for K8S resources through SQL and natural language through a search bar on a web UI.
In fact, recently it is preparing to integrate Cyphernetes as a new search method. I believe this will be a new start!
I'm not against replacing jq/jsonpath for the right tool, they're not the most ergonomic. What isn't clear to me though is why this isn't SQL? It's so nearly SQL, and seems to support almost identical semantics. I realise SQL isn't perfect, but the goal of this project isn't (I assume) to invent a new query language, but to make Kubernetes more easily queryable.
I’d disagree and say that Kubernetes is much more relational that graph based, and SQL is pretty good for querying graphs anyway, especially with some custom extensions.
Graph DBs are generalized relationship stores. SQL can work for querying graphs, but graph DB DSLs like Cypher become very powerful when you're trying to match across multiple relationship hops.
For example, to find all friend of a friend or friends of a friend of a friend:
`MATCH (user:User {username: "amanj41"})-[:KNOWS*2..3]->(foaf) WHERE NOT((user)-[:KNOWS]->(foaf)) RETURN user, foaf`
For SQL based query, you could take a look at Karpor(https://github.com/KusionStack/karpor), which provides powerful, flexible queries across multi clusters, also in the next release we will support Cyphernetes as other query method.
This looks great for scripting. I will say that the query language looks a bit too verbose for daily use — meaning when you're interacting with a cluster to diagnose a problem, follow a job, testing the rollout of something experimental, or similar.
For example, I'd love to be able to just do this as the whole query:
metadata.name =~ "foo%"
or maybe:
.. =~ "foo%" // Any field matches
or maybe:
$pod and metadata.name =~ "foo%" // Shorthand to filter by type
I think a query language for querying Kubernetes ought to start with predicate-based filtering as the foundation. Having graph operators seems like a nice addition, but maybe not the first thing people generally need?
It's not quite clear who this tool is for, so maybe this is not the intended purpose?
re. intended purpose: Initially I started writing this to help tackle bigger problems - stuff you'd normally use multiple nested kubectl commands or write a lot of code for interacting with api-server.
Over time, I developed the shell environment around it and it became a daily driver for me as well. Indeed, there's a threshold where writing Cyphernetes becomes more economical than using kubectl but for doing most of the simple day to day stuff writing Cypher is too verbose.
The Cyphernetes shell has an early-stage feature that allows a syntax like you suggested - there's a tiny "macros" feature that lets you define custom procedures of one or more queries (currently shell only, not supported in the web client yet).
Macros are prefixed by ":" and you could define something like:
:pod condition
MATCH (pods:Pod) WHERE $condition
RETURN p.metadata.name, p.status.phase; // and whatever other fields you'd like
Then use it like this:
> :pod .metadata.name=~"foo%"
So it gives you a tiny way to customize how you do this day-to-day stuff.
Ships out-of-the-box with common stuff you do with kubectl like :getpo, :getdeploy, :createdeploy, :expose and so on - definitely a feature that could be developed further to make this more of a daily driver.
Agree but I'm not sure why. I'm not a mac user so the initial impression is like "this isn't for you, go away". At least add a linux command alongside it!
Even on macOS, brew is wildly inferior to MacPorts; to be fair, brew is “blessed” by Swift Package Manager whereas MacPorts is not, but this is ironic given the guy behind MacPorts both worked at Apple and designed the original FreeBSD ports system.
That's the very weird thing about the little universe within Apple's macOS, there are pockets of very high quality here and there, next to "what the fuck are you guys even doing?", and a Unix core that's been effectively bastardized, abandoned, and frankenstein'd
Instead of Compton a short ride from Beverly Hills it's like the houses from those two wildly different hoods all stacked in a repeatedly alternating sequence.
I think the overall quality level on Macs is leagues above anyone else, even the UNIX stuff (I personally prefer the BSD utils to the GNU ones, which pretty much anyone else running POSIX stuff is using unless they’re daily-driving an actual BSD.
I just don’t get why Apple would hire Jordan Hubbard for their UNIX team, see him implement a very well thought out version of the standard package manager of all time that he also wrote, and then decide to use brew as their blessed package manager for their various open source releases, .systemLibrary in Swift packages, etc.
Kubernetes only runs on linux, so it follows to reason if you care about k8s you should care about linux. My experience is also that good experienced sysadmins often use linux for their own machines as well.
Targetting a tool at macOS users, and omitting linux instructions, gives the impression that the tool isn't targeted at sysadmins or hackers (i.e. at us), but rather at beginners, frontend developers, etc.
Saying it's targeted at beginners because it supports MacOS shows a lot of disconnection with what many DevOps people use these days. The year of the linux desktop has yet to arrive, and Mac is king for people in IT (at least in the US)
I have yet to meet a competent sysadmin that cares much about "desktop", and to the extent they do they mostly seem to invent their own graphical tools, with Tcl/Tk and so on.
Most, however, do not, nor should they be expected to. Homebrew is not a safe or viable package manager, especially when better and safer package managers exist in the Linux ecosystem.
This is fantastic. I’ve always enjoyed the cypher language that the neo4j team created for querying graph data. The connected k8s api objects seem like a great place to apply that lens.
I really really like Steampipe to do this kind of query: https://steampipe.io, which is essentially PostgreSQL (literally) to query many different kind of APIs, which means you have access to all PostgreSQL's SQL language can offer to request data.
* it's super easy to request multiple Kubernetes clusters transparently: define one Steampipe "connection" for each of your clusters + define an "aggregator" connection that aggregates all of them, then query the "aggregator" connection. You will get a "context" column that indicates which Kubernetes cluster the row came from.
* it's relatively fast in my experience, even for large result sets. It's also possible to configure a caching mechanism inside Steampipe to speed up your queries
* it also understands custom resource definitions, although you need to help Steampipe a bit (explained here: https://hub.steampipe.io/plugins/turbot/kubernetes/tables/ku...)
Last but not least: you can of course join multiple "plugins" together. I used it a couple of times to join content exposed only in GCP with content from Kubernetes, that was quite useful.
The things I don't like so much but can be lived with:
* Several columns are just exposed a plain JSON fields ; you need to get familiar with PostgreSQL JSON operators to get something useful. There's a page in Steampipe's doc to explain how to use them better.
* Be familiar also with PostgreSQL's common table expressions: there are not so difficult to use but makes the SQL code much easier to read
* It's SQL, so you have to know which columns you want to pick before selecting the table they come from ; not ideal from autocompletion
* the Steampipe "psql" client is good, but sometimes a bit counter intuitive ; I don't have specific examples but I have the feeling it behaves slightly differently than other CLI client I used.
All in all: I think Steampipe is a cool tool to know about, for Kubernetes but also other API systems.
Steampipe project lead here - thanks for the shout out & feedback multani!
I agree with your comment about JSON columns being more difficult to work with at times. On balance, we've found that approach more robust than creating new columns (names and formats) that effectively become Steampipe specific.
Our built-in SQL client is convenient, but it can definitely be better to run Steampipe in service mode and use any Postgres compatible SQL client you prefer [1].
You might also enjoy our open source mods for compliance scanning [2] and visualizing clusters [3]. They are Powerpipe [4] dashboards as code written in HCL + SQL that query Steampipe.
our project https://github.com/stackql/stackql has a k8s provider which might be of interest here, we implement our own front end SQL parser and expose all control plane routes (and data plane routes in many cases) through overloaded SQL methods, this is not FDW based and does not require a server (postgres etc)
This is way cool. The ability to visualize the k8s object model as a graph and query it as such makes so much sense!
The hottest feature in my mind is applying this in an operator - maintaining state as defined by a simple graph query. It is much more readable, and does so with very little code.
Well Done!
Since you mentioned visualize k8s object model as a graph, I guess you might be interested in: https://karpor-demo.kusionstack.io/, hope this can bring some new possibility to you.
Cyphernetes seems capable of graph/relational logic.
The example on the homepage is literally "give me deployments with more than 2 replicas with pods that are not Running, and give me the IP address of the service they're serving"...
Any idea how to do that with kubectl | jq? Their solution seems elegant to me.
the thing is you'd need 3 k8s queries, one for pods, one for deployments, one for services, then link all of them, and filter... jq helps with the filtering, kubectl can query, but you still need to join the 3 resources to answer the query...
since cyper-based (instead of sql), is the key question whether my k8s data is more graph-like or relational?
adjacent but lots of experts here - independent of Cyphernetes or specific tooling, what are you doing to secure k8s api / kubectl / k8s control plane?
I dunno, Kubernetes has a query language, it's called jq. As in, kubectl get pods -A -ojson | jq -r '.items[] | ...'. Cyphernetes seems simpler perhaps but it's not the 10x improvement I need to switch and introduce a new dependency.
I guess they would say that you have to send the output of that to be inputs of another kubectl command like
$ kubectl logs -n foo $(kubectl get pod -n foo | awk '/Running/{print $1}')
because one of their selling points is "no nested kubectl queries".
I don't see how their queries can be more efficient than hitting the kube-apiserver multiple times, unless they have something that lives clusterside observing lifecycle events for all CRDs and answering queries with only one round-trip instead of multiple.
Or maybe they're selling "no nested kubectl queries" as an experience feature, saying that a query language is more ergonomic than bash command redirection. My brain has been warped into the shape of the shell, for better or for worse, so it's not a selling point for me.
I am firmly in the camp of jq because (a) I am able to bring my years of muscle memory to this problem (b) jq is without a doubt more expressive than jsonpath (c) related to the muscle memory part I have uncanny valley syndrome trying to context switch between jsonpath and jmespath (used by awscli for some stupid reason) so it's much easier to just use the one true json swiss-army tool upon the json those clis emit
