This is really interesting. Copying to the scratch container is powerful and I’ve used it a lot, regularly, but occasionally something comes up where I need to use a more fully featured base to support things. One other downside I’ve encountered with Scratch is no shell so doing docker exec or kubectl exec doesn’t work. Does anyone know a good solution to this problem?
The Kubernetes folks' solution to this is the addition of `kubectl debug` (added as `kubectl alpha debug` in Kube 1.18, graduated to `kubectl debug` in Kube 1.20) as an alternative to `kubectl exec`. It takes an existing Pod and lets you attach a new container with whichever image you like, so that your production images don't need debugging tools.
Also, before `kubectl debug`[0] existed, you could always edit a Deployment and add a sidecar container of `alpine` or `busybox` and enable process namespace sharing[1] get some leverage to debug with.
Yup it would trigger a restart -- that first link details some other options if restarting the workload is absolutely not an option. You can debug a copy, start a privileged pod, or jump on to the node and actually enter the namespace manually. At the end of the day all these containers are just sandboxed+namespaced processes running on a machine somewhere, so if you can get on that machine with the appropriate permissions then you can get to the process and debug it.
Of course, if you're in a tightly managed environment and can't get on the nodes themselves things get harder, but probably not completely impossible :)
We drop a statically compiled BusyBox binary on images like that as "sh". If we need more we can symlink to it in /tmp at debug time (or just call it directly). It strikes a good balance between slim and debuggable.
An emerging solution I've been investigating is using Nix as a build system for docker. The syntax is fairly lightweight and you can build containers that are scratch + coreutils, or whatever else you decide to put in there.
I was originally turned onto this by a post on the repl.it blog about using Nix this way.
You can also do this with other build systems or using weird docker file hacks by hand.
If you run a single process in a Docker container, have it output logs over STDOUT, and have no meaningful way to interact with it other than shutting it down, you don't have much need for a shell. It can't do anything in the container anyhow. By contrast, if you've basically got a full OS in there, then yeah, a shell is really useful.
I would say most of my Go projects involve what you described (logs to stdout, simple interface) but I often find myself needing a shell to debug my docker build, like ensuring that files have ended up in the correct place, and have the correct contents, within the docker container.
If your stack involves running containers on a Linux machine, you can use `nsenter` to use the debugging tools on the host OS to debug the processes within the container.
In the article they statically link in SQLite, It’s the reason the Go program can just be copied to a scratch image and still work.
You only need gcc during the build fase, that doesn’t change just because you run without Docker.
There’s nothing preventing you from statically link C programs either and have a single binary to deploy either. You just have to rebuild everything every single time security updates are a available. But that no different from Docker images, they need to be rebuild constantly as well.
