It's compatible with cgroups v2 unlike the standard Docker. If you're using Fedora, you have to add a kernel parameter to Grub to use cgroups v1 instead.
RedHat seems to be pushing a standard ecosystem for Linux: systemd, Wayland, SELinux, GNOME, and now maybe podman. I've been on Linux for a while; it's a welcome change from all the fragmentation I'm used to. Whereas others try to work around the kernel and implement their own things in parallel (see: Canonical's AppArmor, LXC, OpenZFS), RedHat just goes with what Linux already has like SELinux/cgroups v2/btrfs, which I think is more likely to last and just feels better. If RedHat goes away, I'm fine since I'm ultimately only relying on Linux features. If Canonical goes away, then I'd have to switch to a different stack. That's probably why government, enterprises, Amazon, etc. still prefer RedHat.
What you are noticing is the RedHat “upstream first” policy, and it’s very practical. They try to minimize RedHat specific modifications and instead contribute improvements directly to the original projects.
This reduces the amount of code they are responsible for maintaining, and everyone involved gets the benefit of open source collaboration.
More like, Redhat/IBM defines what Linux has simply by paying most Linux devs, and of course they take advantage of it in userspace and sys management.
cgroups/namespaces, systemd, gnome (and it's integration with systemd) plus others are two-sided swords: one one hand they create new functionality, but on the other hand the price to pay is concentrating Linux know-how in one place, and greatly diminishing portability of Linux apps vs other Unix O/Ses and even other Linuxes that don't want to go with the program of absorbing ever more functionality into kernels and system frameworks for no real reason other than monopolization.
Unix was designed as a simple portable operating system in very short time. With 30 years of development, the situation today could also be interpreted in such a way that Linux devs just can't stop to add code. Time will tell if Linux can be maintained if the original generation of devs step down. I'd feel more comfortable if we've let kernels stay minimal rather than becoming kitchen sinks. Apart from better portability, this would've also enabled younger devs to start from scratch rather than having to maintain daddy-o's monstrosity.
>and greatly diminishing portability of Linux apps vs other Unix O/Ses
Thank's god for that! An OS should build on its strenghts, not be a least common demoninator for portabillity's sake. If that was ok we might just as well just use one OS
Indeed, for me the best UNIX experiences that have been created so far have been NeWS and NEXTSTEP (macOS), and I wouldn't mind if Linux would just standardize on one stack as well.
As a 23 y/old younger (well, young-ish) dev, it strikes me that this isn't anything new. Old commercial solutions since nearly the beginning have been predicated on proprietary solutions specific to some vendor's UNIX based OS. HP-UX, AIX, even back to the really early commercial releases of v7 all seem to have a common theme of having that unix core but with a lot of nice features added on, typically with pretty insane and convoluted frameworks. Tru64's clustering, AIX's management interface, all the competing UIs and management interfaces. All nice features deeply integrated into a proprietary solution as a form of vendor lock-in.
To your point, it seems like eventually this ends poorly for the incumbents, when they become too bloated and too difficult to extend or modify, since you have to work around some over-engineered high-availability process clustering to touch anything.
In Linux's case, I think the kernel itself is actually fine, but the user land is increasingly tightly glued to an increasingly convoluted set of RedHat stacks.
I mean, it always has -- RedHat made the modern posix threads library, Linux's PAM, and several other core parts, but at least those were pretty modular and "unixy", by comparison to the monolith being developed now.
Anecdotally, I've worked with developers at Redhat and Canonical, and the Redhat developers had passion. They really believed in open source and the linux community. In comparison Canonical seemed like Just Another Software Company to me.
I think governments prefer redhat because they are consistent and they accept gobs of cash to promise security patches to software that are past their support window.
Governments also prefer Redhat because they early on and continuing today have shown a commitment to the market by investing in getting their products certified to meet government required standards including developing the required capabilities when not already available in the Linux ecosystem, developing documentation and software to assist deploying systems in a manner consistent with government security requirements and participating in conferences, working groups, interoperability tests and other government targeted or sponsored forums.
Some other Linux vendors now also do so but started much later and even now do so to a much lesser degrees.
I don't know about the gobbles of cash, but I'm pretty sure Canonical supports Ubuntu releases way past EOL for paying customers - they call it the ESM (Extended Security Maintenance), which 'provides important security fixes for the kernel and the most essential user space packages in Ubuntu'.
Depends on your definition of support I suppose. Redhat employs so many more low level engineers they can generally get you a z stream patch or something to fix a legitimate bug. Canonical is an exceptional marketing company, but I've found their engineering to be lacking when it comes to enterprise. As someone who's worked as a linux monkey for companies who pay Canonical for support and for companies who pay Redhat for support, I've always found the Redhat support to actually solve the root cause. When we filed what we thought were legitimate bugs with Canonical, the common response was just to "upgrade to the latest version of Ubuntu". When we asked them to confirm that doing so actually fixed our problems we just got crickets.
It tends to also be in the opposite direction, big expensive enterprise software is supported on redhat forever and ever. And the issue isn’t just security updates, but full software and hardware support for the same stack for many years.
Lots of people don’t want to recreate their stack every few years for the latest cool thing.
Both of you are right. Red Hat pays for Fedora in a lot of ways, but does not control it. There are obviously a lot of personal and professional relationships that gives Red Hat influence over Fedora, but the purse strings are never used to control it.
If you want to use btrfs on RHEL8, you still can: go to https://elrepo.org/ and install elrepo-release, then install kernel-ml from their -kernel repo and btrfs-progs from their -testing repo.
Running an unsupported kernel doesn't mean that support won't help you at all. It just means they won't help you with kernel-related issues. There's still a lot of value in support for the rest of the system.
At my company, we run our CI/CD (Jenkins) using the Docker-in-Docker paradigm to facilitate easy maintainability of the CI itself and allow us to run containerized builds.
When we shifted to RHEL 8, we attempted to move this over to Podman and it went miserably (this was back in November 2019). The main reason being is that podman-in-podman doesn't work and had bugs (at least back in Nov 2019). Maybe it fixed now but this was our experience. We ended up doing quite a bit of analysis on podman only to conclude it's simply not there yet relative to docker (ecosystem and ergonomics).
There are quite a few corner cases that docker quite simply supports out of the box beautifully that podman doesn't support or just has bugs.
I like the what the project is trying to solve by being daemonless, but this is not as simple as a drop in replacement for docker that RedHat markets it as (alias docker=podman).
We ended up sticking to docker professionally and personally, I am still using docker over podman. The ecosystem and ergonomics are just far too nice to give up over podman.
Maybe you are just too far into docker. I noticed that a lot of default workflows (needlessly) depended on docker running with privileges. One big reason for that seem to be Mac users that only know docker from inside a VM. However, if you think about what you're really needing for CI you will easily see that docker-in-docker gains you nothing. You can as well use plain docker (or podman). The same holds for privileges. No CI operation should need privileges, if only for the reason that it should never alter the CI system itself.
I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.
Okay, I want to build a container image using gitlab CI, which runs builds in docker. How would you like me to build an image without using docker in docker, or buildah in docker?
We use kaniko[1] in Gitlab CI and it’s working great for us. It’s annoying the kaniko image requires us to specify the entrypoint. There’s some peculiarities with Dive [2], but otherwise it’s been a very easy migration.
Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO.
Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code?
IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too.
I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable.
Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.
People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF?
> I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.
I want this framed or sewn onto a pillow or something.
It's amazing what we can build, it's baffling what we have built.
> Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler
No, the idea is to build and push images you can test directly afterwards in the same conditions. With cache and such, build times shouldn't be too long
Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally.
If you have long CI times, that can hinder development productivity and should be improved as much as possible, or a local replica of the CI needs to be created.
It is really working for you?
Presumably you setup kaniko to build the images first that then run the second part of your pipeline. To run the images, you need to tag them with something that you use inside the gitlab-ci.yml, right?
Now what happens when two people push code that make changes to the containers at the same time?
Your problem is the assumption that a gitlab CI process requires a docker image. If you do the following, you are fine:
* Switch to a shell runner
* Put the CI dockerfile into your repo
* Provide an entry script for CI that builds the container on-demand (and manages caching/cleanup) and then runs the tests/whatever inside that container
The point here is that docker/podman provide you with everything you need as long as you have full control. By using gitlab's default CI, you relinquish this control.
But how would you run integration tests between multiple docker containers? We launch our services in these docker containers and do some integration / e2e tests, and this very much requires Gitlab CI to launch docker containers while inside a Docker container.
After using dind for some time we chose to just mount /var/run/docker.sock and keep using the host machine’s docker instance (mostly for the cache), but all in all dind was working fairly well.
To be honest, to say “you shouldn’t be doing that” is missing the point; one should be able to do anything they want. In my opinion, the root cause here is docker’s client/server model, which is fixed by other container runtimes such as podman and rkt (which unfortunately is deprecated). One should be able to just launch containers as if they were just another process.
To the contrary. You should never expect to be able to do what you want with a system provided by someone else. Gitlab has its design decisions in their default setup and you need to take control to cater your use case. This herd mentality of "everyone is doing it this way" is really fundamentally problematic.
Nowhere in my post am I saying "everyone is doing it this way".
You started out with asserting "you're too far into Docker", we bring up valid use cases for docker-in-docker, and then you saying "This herd mentality [..] is really fundamentally problematic" is really not adding a lot to the discussion.
No one brought up valid use cases for docker-in-docker. They brought up the issue that gitlab mandates docker as an interface (which I totally understand, btw.).
For instance the "how do I compose multiple docker containers" is trivial when you can just execute a script that runs docker or podman. If you really want, you can use docker-compose.
That is exactly my point. Gitlab uses docker runners because it is much simpler for them. But why should you be constrained by what's simpler for gitlab?
Our org just adopted Makisu[1] which works beautifully in our GitLab CI pipelines. After a week of setup and fiddling with the settings, we were able to migrate all our build jobs to Makisu and haven’t looked back. Build times are great too, especially if you set it up with a Redis cache.
Why do you say that docker-in-docker buys him nothing? It's not obvious at all and you go into no detail whatsoever to back up your opinion.
In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.
It is because to build the image using docker requires the docker client to talk with a dockerd daemon, so one has to configure the client to access the dockerd which allow untrusted code to run as root in the host.
Docker-in-docker is a workaround to make docker work in CI.
Basically a security nightmare and bad design that podman doesn't have.
Any build script can do serious damage to the environment it runs in. Before docker, you'd have to create a new VM from time to time because the build agent had rotted away or died in an altercation with a bad build.
Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it.
I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.
You do know that spinning up a new VM only takes a few seconds? With projects like https://firecracker-microvm.github.io/, the difference between launching a new Docker container or a new VM is negligible.
This works great if you own or rent the hardware, but most cloud providers don't allow nested virtualization.
The cost is not spinning up the vm, it’s maintaining the images. Docker composability reduces the combinatorics problem to a dull roar, and democratizes some of the maintenance effort. You want an image with the bug fix from the latest point release of python? And you need it by noon? Knock yourself out.
Although there are tools to convert docker images to vm images. I expect if I were running community CI infrastructure, getting really familiar with those would be high on my priority list.
The other option that works really well in a single user environment is to bind to the runner's Docker daemon. That way builds run as siblings of the runner's daemon rather than as children via docker-in-docker.
The huge issue with that is security which is why it's only really practical for a single user or a small group of trusted users. A secondary issue is that (I think) builds can't run simultaneously because they can trample each other when tagging images (since all images are on the runner's daemon).
If I had to build a Docker focused CI system I'd think about using Weave Ignite (AWS Firecracker) to spin up VMs for runners with the Docker socket bound like described above. That way you get all the convenience of binding the Docker socket, but the isolation of a VM that gets thrown away after the build step (or pipeline) finishes. That idea also fits well with local running / debugging IMO because you can bind to the Docker socket on your development workstation (assuming you're not running a large build of parallel tasks which might be an unrealistic assumption).
For us it’s a matter of the CI tool fetching the source code for the docker image, then running docker build, and not necessarily immediately. So you have ‘docker build’ happening toward the end of a set of other tasks. Which I’d really like to have running on a fresh VM or container.
You could separate those into two builds, but the reason they are together is so people think about deployment, and in case any structural changes to the code need to coincide with deployment changes. For instance, breaking changes in APIs. I need a new version of tool/library and I need to change how I call it.
Kubernetes is a consistent management api for linux (so then you don't need to interact with iptables, mount and all that "hard stuff").
I don't think kubernetes is a solution for context of building an image (a rootfs tree into a .tar.gz file).
Unless you are using kaniko which extends the kubernetes api to add the capability of creating images, but that is handled by kaniko itself via the same api.
I was probably unclear. Kubernetes is a good solution for managing containers (obviously). I use it for CI and it works very, very well, though the CI tools still have more features they could add with the integration.
>> beastly 5G docker image
> my beastly 12GB image that even includes Matlab wants a word with you
Perhaps in the next 10 years we will be rediscovering packages. :P
If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
> If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
True, that.
To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
Well, to be fair, it is packages - I'm just using Docker (for this section of our stack) as a different sort of VM, essentially. It runs a service manager and a VNC X session, for chrissakes ;)
Our images are 35GB, and I've spent much of the last two weeks breaking up files so we don't hit the 8GB per file limit, and my next week will be trying to avoid hitting the per-layer limit.
Docker and Nix are mostly orthogonal technologies though. Nix is an excellent build tool while Docker is useful as an universal software distribution format. They're really useful together because Nix is actually good at creating compact Docker images. But of course, if you don't have to worry much about distribution, it would be much nicer to stick with Nix as you've mentioned.
Well, in the case of Mac, basically performance, as you still need linux libs, kernel API and everything. But it's much lighter-weight than something like Virtualbox, VMWare, etc.
In Linux iirc, a hypervisor can share such resources with the host system (since they are both Linuxes).
We're working hard on docker-compose support. With the new Podman 2.0 REST API with a Docker compatibility layer, we are close. Not quite there, but definitely strategically on the roadmap.
Apologies for the patronising comment, but do you really mean that? Docker in Docker works but is intended for the developers of Docker to debug Docker itself. Usually for running Docker from within a container, you just hook up the Docker client to the TCP port of the Docker daemon running outside the container, which isn't strictly Docker-in-Docker.
I ask this in case you're trying a wildly use case (if you really are running true Docker-in-Docker), or are making an unfair comparison (if you're just using the usual Docker client in container to daemon on host). In the latter case, I must admit I don't know what the idiomatic alternative would be for podman, given that I know nothing about it except that it's daemonless (and even that I only learned by reading your comment).
I think what most people mean when they say docker-in-docker is in layman terms. You simply just a way to do "docker run" from within another container. Whether that's done by actually running docking daemon inside or mounting something from the host or whatever else i don't really care. I just want it to work.
Sure, there are some details you might want to control, like image caches and such being shared with host, i just find there is a lack of documentation and best practices of how to do nested docker, if that is even intended to work or if mounting docker.sock is an unsupported hack. Most information found about this is scattered on shady blogs.
For the examples of CI brought up the use cases are fairly obvious, you have a jenkins installation with x plugins installed - running as a container. Within this jenkins you are building multiple different projects which all require their own respective image to build. As a project developer here i don't even want to know if this jenkins is a bare metal, a vm or a container. Here docker is more used to bundle all the dependencies, not for strict security with perfect containerization.
If you mount docker.sock you're building on the runner's daemon so you have to worry about builds interfering with each other. If you use docker-in-docker you get a clean environment every time.
If you don't manage your own CI server then that is not an option is it?
I also never said that it requires two levels of Docker or that it requires DinD. I was responding to the general question of the parent who was asking a question of someone who was running DinD. I responded to the GP below that Kaniko also solves this problem so clearly I'm not advocating for running DinD or that this is even needed.
As mentioned elsewhere in the thread, this does not require DinD. You can connect to the host’s Docker daemon. (Other interfaces such as K8S are also much safer.)
Yes it is. The Docker socket - /var/run/docker.sock is owned by root and so anyone that has access to it effectively has root on that host. User namespace mitigate this to some degree.
Yes and I am one of those people who mentioned elsewhere that this does not require DinD. Please reread the question I am actually responding to which is a general questions "What kinds of tasks ...?" The parent is asking what 'problem is the DinD solution is solving for. You also can not connect to the host's Docker socket if you do not manage that host where Docker is running in the first place.
“kind” is a example in my day to day, how do I run an ephemeral kube cluster in ci to deploy and test my app against without dind/kind? I want N clusters running on 1 host to support many CI jobs.
Also if your application is shipped as orchestrated containers (like docker-compose), or as multiple containers in a 'pod' (e.g. sidecars), you may want the ability to run containers from containers as part of CI.
As of Molecule 3.0, at least -- they didn't in the past.
A while ago, I, unfortunately, decided to add "proper tests" to all of my Ansible roles and decided to use Molecule (2.22, at the time, IIRC). As I don't use Docker, I was using "lightweight VMs" I had created (w/ Packer, converted into Vagrant boxes) with VirtualBox for all of this testing.
I spent I don't know how many hours across several days learning the "toolchain", getting everything setup and working properly, adding full test coverage, and so on. Not long afterwards, they released Molecule 3.0 which required using Docker. :/
We use docker-in-docker as well to deploy our Jenkins workers. It has it's pros (running Ubuntu 18.04 and Ubuntu 20.04 workers side-by-side and ease of pushing out an update) but it's fragile and confusing. Things can get really hacky when you need to start sharing volumes or injecting additional configuration.
I'm much more confident with packer now though. Next time I do any major work on our Jenkins infrastructure I'm ripping out docker-in-docker for the workers and replacing it with packer built images.
> but is intended for the developers of Docker to debug Docker itself
Heh. We use DIND with docker compose to have a container which has KUBERNETES inside. And it even works.
How's that for a wild use-case?
EDIT: That's done to create a local dev environment, with K8s, localstack, infra, etc. Instead of having multiple machines or deploying everything outside containers.
I also needed to run docker commands from within a container (Jenkins container executing docker cmds on the host it's running on). I don't know how common this is but I had the same use case.
I do a lot of build automation work, and running containers within containers would be super useful for me.
That said docker-in-docker doesn't work without running privileged or forwarding the host port.
It's a non-starter for me, there are the obvious security problems, but also practical non-security issues.
Forwarding the port causes encapsulation issues, a build job can finish leaving stuff running, can also interfere with other jobs on the system.
Using privileged containers isn't an option when using things like ECS fargate.
There are some reasons to use docker in docker beyond debug docker itself. For example, in Jenkins case, select specific proxy settings for the daemon for a specific job without affecting other jobs in the same node.
I had the same experience, and I'm not really sure why I can't use docker-compose with podman. If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal?
I did try that and it doesn't work, and podman compose isn't as useable. So I switched back to docker.
>If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal?
Podman is (mostly) a drop-in replacement for docker. However, docker-compose is a separate package from docker that requires explicit installation. So too is podman-compose separate from podman, though unfortunately it still needs more work.
I can explain this. Because docker-compose talks to the docker socket and uses the docker protocol. This was the last piece that we nailed with podman. As of Podman 2.0 with the new REST based API with a Docker compatibility layer, we are very close to just being able to use the docker-compose binary to talk to a socket (ran by systemd) which fires up podman.
It doesn't quite work yet today because we are still implementing some of the REST verbs, but it's close. It's definitely a strategic direction for the roadmap. Stay tuned.
Spend too much with Keycloak lately - it's more like a framework with some basic batteries included and you'll hit into corner cases pretty fast. If you are fluent with Java, their templating engine and all the other funky stuff it's a powerful tool - but i.e. their REST-API is bad. List all users including groups? That'll be an additional HTTP call for every user.
Their tools solve complex problems and are free to use - so I'm thankful that they exist - but I can't help but think that there is some lack of elegance&design that causes lot's of complexity - there is no "unix philosophy" to abuse different tools or components to solve problems - it's more like either you'll use the high level APIs with certain non-obvious assumptions (i.e. worked for us, good luck :) or feel free to hack on it if you grok our complex low-level frameworks and libraries... went after a NetworkManager bug once and it was a tour de force between c, glib, dbus with zero documentation. systemd and Keycloak feel very similiar. Powerful if you fit their usecase - horrible if need to tinker with it. But to be honest I've got no idea how to solve these complex problems otherwise. It's probably the best we can do at the moment. Or are there any non-cloud/non-sass solutions that actually have all the features?
This might be biased on my part but experiences like this is why I tend to openly avoid most Red Hat projects in general. There are definitely some great projects like RHEL, Ansible that are nice to work with.
Even professionally, I tend to steer clear of immature Red Hat projects.
We evaluated Keycloak but went with a vendor solution. OpenShift I believe was also evaluated at my firm and hit a dead-end.
Openshift is a great solution for a k8s setup on prem. From my experience it doesn't bring a whole lot when you're already on the cloud. All the additional services openshift provides, are already available as part of the major cloud offerings.
Openshift years ago at least (last time I looked) - SHUDDER! I couldn't believe it was what big business / enterprise was standardizing on as best in class. It looks like it's now Kube / Docker connected, but back then I think it did it's own thing?
The cost in time / $ / config to maintain and operate at the time was not pretty (all this no doubt has changed a lot).
They gave you three pods for free until they moved to Openshift 3. I ran an entire startup five years using them without paying a single penny. A nice Heroku alternative. (Iot control server, so basically no load)
Openshift was very generous to startups, I can back that up as well.
It always seemed like added complexity to me in exchange for free hosting. I liked the idea in concept more than in practice. But we also weren't big enough to really justify it so eventually we switched to simpler VPS hosting.
Your life will be a lot less painful if you abandon docker-in-docker. The Jenkins solution is to simply execute containers using the host's Docker daemon, or via a container orchestration system which you call out to.
The latter is the best option as it's much more scalable and doesn't require ugly proprietary hacks. Docker in Docker requires vertical scaling and more complex management for intermediate states/maintenance.
Podman probably isn't ready to replace Docker yet, but rootless containers are the easy-mode for federation of clusters. If you can supply the other features you need (and I think most can) it's probably worth it in the long run.
If you are using Docker-in-Docker, you may want to checkout the new Sysbox runtime (find it on Github). It's a new type of runc that sits below Docker and creates rootless containers capable of running Docker, systemd, K8s, etc. All you have to do is "docker run --runtime=sysbox-runc" <some-image-with-docker> and you'll get a docker daemon that is fully isolated from the host. It's a great way of avoiding privileged containers or mounts to the host docker socket.
I love the idea of Podman being daemonless and (potentially) rootless. Regardless of the presence of security flaws in dockerd, the Podman architecture just feels cleaner.
And the CLI compatibility is great. Until it isn't. At work we switched to Podman for a small deployment because Docker didn't yet work with cgroupsv2 and many hours were spent debugging Podman-specific issues. In the end switching to cgroupsv1 would have been significantly less work.
Therefore claiming you can `alias docker=podman` is a bit disingenuous. You can, but only if you don't do anything Podman can't handle and what it can and cannot handle isn't immediately obvious.
All that said, I wish the project the best and hope it reaches the maturity where this alias actually does work.
I'm afraid I didn't and I cannot really recall, but many were related to networking. I think one was that I couldn't have two containers on the same network listen to the same port, another one was related to Podman not playing nice with nftables, but the specifics elude me.
EDIT: Upon further reflection, I think Docker doesn't really work with nftables either, so that one isn't on Podman. It just so happened we made that switch at the same time. Regardless, there were other problems. I'll check to see if I can find any records of the problems later.
I was just about to complain about another thing to learn. Then I saw you can do "$ alias docker=podman". Just want to acknowledge the importance of that work. Making things compatible is both boring and a pain, but it's a door opener for people like me who refuse to learn your new API because I know a decent one already.
One thing I want to point out though for anybody not familiar with the differences between podman and docker, for the most part alias docker=podman will "just work" except for these situations:
1. docker-compose. podman-compose attempts to cover this but I've heard it's not quite there yet
2. Mounting the docker socket into the container. Podman is daemonless which means that won't work. There is work going on right now to allow Podman to be driven in a similar way if needed, but I recently tried to set it up and hit a bug[1]. CRI-O brings a daemon, and it is used extensively in Kubernetes and OpenShift, but not so much outside of that.
So far, it's just a less functional docker and didn't bring much to the table.
I don't feel the need for daemonless as docker has been working fine.
I've swapped back to docker on Fedora and CentOS by forcibly installing docker on those unsupported platforms because podman-compose doesn't work, containers just didn't work like expected in several situations (forgot detail) and portainer wouldn't work against it.
Either they should have sped up the podman development or shouldn't have deprecated docker at this point. Maybe RedHat just didn't like a third party taking the stewardship in the container business.
Unless you're optimizing for stagnation, this really is a terrible approach to reaping the benefits of innovation. If your prereq to trying something better is that it needs to mirror something that's "decent already," you'll spend most of your life stuck on the same plateau.
History is full of technologies that are superior to existing options on one metric or another and yet failed to take over due to switching costs. Attacking someone as “optimizing for stagnation” is a really bizarre way to view their reaction to a product offering a smooth on boarding process.
Yeah, you're right. I should make the time to learn more stuff and never make excuses for why I don't.
On the other hand, I don't think you fully understand where I'm coming from. "Optimizing for stagnation", "stuck on the same plateau". Idk, I just want to run a container. I'm totally fine being on the same Docker plateau if something like Podman makes me re-learn what I already know how to do.
Unlike Git, which, from the start, was different and BETTER than SVN, can you really make the case that Podman would be BETTER if it didn't try to follow Docker conventions? If not, what are we talking about here?
We've tried this (podman 2.0.5) and have hit some really really annoying bugs:
- Build layer cache doesn't seem to work. If I rebuild locally with podman, it correctly detects cache hits and the build is fast. On our Jenkins server (RHEL 8) with podman 2.0.5 it doesn't. It randomly doesn't cache hit, causing builds to take 20x longer than with docker CE.
- Podman is insanely slow at building images in general. COPY emptyfolder/ /emptyfolder/ takes 2 seconds. We have dozens of things to COPY and it's stupid slow compared to docker CE. Buildah doesn't seem any better.
- Systemd integration has bugs. If you use the default generated systemd unit file, it does not kill processes when exiting and leaves them dangling. Even after removing the strange KillMode=none it says to put in there, it still leaves processes dangling. Podman sometimes loses track of the container. It will list nothing in "podman ps" but the processes will still be running.
Not using --cache-from, though I did try the various caching command line options to try to troubleshoot. Just a vanilla Dockerfile with a few RUN commands and COPY commands. It would hit cache until about the third layer, and then would cache break every time.
On my local computer (Arch) podman is v2.1.1, which seems to have whatever bug I was hitting fixed.
So I guess my complaint isn't about podman specifically-- It had bugs and they were fixed, and that's great. But I hate that RHEL 8 touts it as a docker replacement, and won't carry docker in their repositories, when the version they have in their production releases is so broken.
We eventually sledgehammered docker CE's CentOS repo into our RHEL 8 jenkins server and now everything works perfectly.
Running podman on the production webservers seems to work okay though-- apart from the process killing problems.
I've said this so many times, but we wanted to carry Docker in RHEL 8, but we couldn't. We would have been forced to use Moby and it's not a well contributed to project anymore. The whole Docker contributor community has fallen apart, and Red Hat needs to be able to support a container engine until the mid 2030s.
You can get access to Moby in Fedora, but it just wasn't viable to include docker in RHEL 8 for both legal and community risks.
I find the podman integrates with systemd well claims a bit dubious.
Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background.
I found this very surprising. I still don't understand why they made that choice.
At least they do play nice with the whole "systemd owns the Cgroups tree" story. A thing that was always a bit painful with docker.
Fun note. Systemd-nspawn actually can run OCI containers directly as well these days. However I'm not sure if it's feature-complete
If you are running podman rootless it will use the fuse-overlayfs driver which comes with various limitations and is much slower than Docker:
* you are limited to 1024 FDs for your entire container. So running make -j40 usually ends up with various errors due to running out of file descriptors. You can of course raise the limit for your own user, but this may not be trivial on a shared system.
* getting podman to obey a raised user FD limit is non-trivial on CentOS 8. podman is not well supported on CentOS, you need latest Fedora if you want the various bugs/limitations fixed.
* fuse-overlayfs is a single process for the entire container and quickly becomes the bottleneck for any IO intensive operation (e.g. don't try running AFL inside podman, it'll be way too slow and peg the CPU at 99% running fuse-overlayfs).
Fuse shouldn't be necessary though if I could just give it access to the files that my user has access to (of course you'd loose the snapshotting/committing ability), perhaps using -v would improve performance?
Using docker on CentOS 7 was a lot easier (and faster) than using podman on CentOS 8.
Hopefully most of you who have had issues with Podman have opened bug reports or issues.
We are working to fix incompatibility bugs and getting out new releases all the time.
As far as docker-compose support, in Podman 2.0 we have added APIV2. This is a socket activation REST API. This API has a compatibility mode that implements the "Docker API" meaning podman can be setup to listen on the docker.sock and launch containers. We also have the more advanced Podman API for support of concepts like Pods. The API should be able to work with docker-py based scripts and Compose. We are getting lots of community support in fixing up our inconsistencies.
This is a fully open source project and we love to get contributions.
This is incredible news! My only reason for using docker still are images that have you mount in the docker.sock and they introspect on it (such as jwilder/nginx-proxy). The day I can move over completely will be a good day.
I always like to chime in and recommend people check out Singularity[0] containers if they haven't before. It's more common in academia because you can run containers without privileges which is nice in HPC environments. The containers themselves are simple immutable flat files that can be easily copied around, backed up, etc. There's also no system daemon. It's nice knowing everything you need for your container system exists in the containers themselves and a single singularity executable. No volumes, remembering to append --rm to avoid dangling containers, wondering where the actual images are stored, etc. Of course there are tradeoffs but I like them for certain use cases.
* It runs containers as the same UID as the calling user, so directories mounted into the container can't be polluted with files belonging to other UIDs, as Docker tends to do
* It's easy to get it to only pull images from custom registries. With docker, this requires some fiddling (at least last I tried).
The only thing missing is Debian packages in Debian Stable (currently available in Bullseye aka testing).
I find the podman integrates with systemd well claims a bit dubious.
Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background and do the process supervision.
I found this very surprising. I still don't understand why they made that choice.
For anyone not using Podman just because lack of docker compose like functionality, there are multiple alternatives.
* Podman's official stance is to use Kubernetes YAML. The provide tools to generate this as well as convert docker-compose to kubeyaml
* podman-compose is not official, but it works well. you have to note that its not 100% compatible with dc but a lot of dc files run out of the box without modifications
* ansible with podman
* i personally use the podman cli with makefiles to get docker-compose like functionality
* also, podman v2 supports a docker comptaible rest api so its possible that docker-compose can be modified to support podman in the future
For my stacks, the only thing i can't get working is routing a specific container through a host defined network (not a container). I'm certain podman can do it, i just haven't had the time to "fiddle" with it.
With docker (compose) i just define an ipam network (external) matching the host adapters network, and connect the containers to that network, and the spice begins to flow.
> One of Podman’s greatest advantages is its complete CLI compatibility with Docker. In fact, when building Podman, Docker users can adapt without any significant changes. For example, you can use the alias command to create a docker alias for Podman:
`alias docker=podman`
Wow that's a great adoption hack; just alias another program as another
I like the daemonless architecture a lot, but until there's a quick and painless way to install it on OSX and Windows developer machines, you're going to see very limited uptake.
I actually like the idea of Docker as a better systemd (or rather, the interface is better). No bespoke file format, programmable API, no need to google for the right journalctl switches, and then of course the advantages of containers and images over processes and system packages. I’m not suggesting everything should be a container nor that docker is the ideal implementation, but it certainly points in the right direction.
> I actually like the idea of Docker as a better systemd (or rather, the interface is better). No bespoke file format, programmable API, no need to google for the right journalctl switches, and then of course the advantages of containers and images over processes and system packages. I’m not suggesting everything should be a container nor that docker is the ideal implementation, but it certainly points in the right direction.
Docker was created to give an extra life for legacy applications depending on outdated packages, the concept of gluing all dependencies withing a compressed rootfs and shipping that, with low effort.
Docker became popular because of that, but seems that did more harm than good for the ecosystem in terms of security, that why podman might have a great future. The docker interface is bad and comparing that with systemd is quite a stretch. ;)
Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.
Aren't containers just processes with namespaces and cgroups?
> The docker interface is bad and comparing that with systemd is quite a stretch. ;)
I don't know, systemctl is always a pain to interact with. The Docker CLI is pretty intuitive--you have different resources and different verbs for interacting with those resources. This includes logs. No need to use a separate (and confusing) tool nor dig through man pages. I'm of the philosophy that it's better for a tool to be intuitive than obscure-but-has-manpages.
> Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.
I guess I meant that INI isn't a standard file format--different parsers behave differently, and structured data is often coded in strings in some bespoke format.
> Aren't containers just processes with namespaces and cgroups?
My mind isn't made up that containers are the ideal process unit, but I do like some things about them (and the container ecosystem more generally). That they come with their dependencies bundled is pretty nice, but I think the toolchain needs to improve to mitigate security concerns and so on. Something like Nix would be helpful, but Nix also has its own problems. Again, it's directionally correct. In any case, users don't need to be managing their own namespaces and cgroups.
But systemd has all of those things. Everything in systemd can be orchestrated with dbus. In fact that’s all systemctl really is — a dbus client. And the file format is ini. And shell style environment files if you want.
And systemd actually comes with two image launching systems systemd-nspawn and systemd-portable. And then with systemd-machined you can add software that needs virtualization too.
The interface to journald is more complicated than it should be but it’s also really powerful — docker logs doesn’t hold a candle to the kind of filtering it can do.
Overall systemd is a stupidly powerful and featureful supervisor compared to Docker. Just the dependency management alone should demonstrate that. Then you get mounts, swap, socket activation, more powerful restarting policy and the whole suite of isolation and security features.
> The interface to journald is more complicated than it should be but it’s also really powerful
I thought the point of logging to stdout (i.e. docker logs) is that you just take that output and dump it to another server for processing and filtering.
journald/journalctl seem to be a solution a few decades late to the party. For a single user machine or a single app prod environment, I would take a plaintext log any day of the week. At least I can remember how to grep the damn thing. And then when you get to a distributed system, what's the point of journalctl? You would hopefully have all of that logging aggregated together in one place with a much nicer web interface for it all.
Exactly this. I just want a stream of logs that I can process with ordinary unix tools without jumping through hoops. I don’t need any extra power for its own sake.
Not a huge deal, but I have to google it every few months because I don’t use it very often.
Note also that if I just want the log stream, I have to pipe through less to get the full log messages. There’s also a flag for it, but I can’t remember my workaround is easier than digging through man pages.
Again, no big deal, just friction. Like everything in the systemd ecosystem—everything is manageable, but it’s tedious.
How are you using Docker that doesn't need a bespoke file format? You're building from a Dockerfile and either scripting around the docker command or running docker-compose.yml
Like I said, I don’t actually want docker or docker compose, but they are directionally correct. Notably YAML over INI is a nice improvement. Also, that the docker daemon has a real API and not just dbus.
Well, Docker has an API and its own file format (the Dockerfile), I constantly have to look up Docker switches as it's not a tool I use every day and it's not at all intuitive, and I'm still unclear why people love this layered/stack image approach given how unbelievably slow it makes everything. Apps that would once have been a 50mb tarball are now routinely a gigabyte of downloads, perhaps because whatever random version of Ubuntu I downloaded previously is now slightly different? Who knows.
There is, frankly, a lot of appeal to me in a simple INI file that starts up something I unzipped.
Last time i did look into docker it couldn't even tell if a service is up or still starting and there were all kinds of weird hacks around that like "is something listening on the configured port". Did that improve?
Systemd struggles with this as well. While a service can easily tell you if it's "up", sometimes the only way to determine if it's "available" is to check. Keep in mind that the sentinel for any given service will be different. If you think about it, how could any external monitoring service determine when a service under it's purview is available without actually checking?
Maybe it already has something like this, but I feel like systemd could provide an API endpoint for applications to send a simple status when they are "ready" - at which point it would be up to the developers to provide that at the right time.
From the link: "systemd-notify may be called by daemon scripts to notify the init system about status changes. It can be used to send arbitrary information, encoded in an environment-block-like list of strings. Most importantly, it can be used for start-up completion notification."
I was just thinking the same thing, and was going to ask if there was anything like Docker Desktop, something to make it simple to use on Windows/MacOS!
I find it somewhat surprising that we haven't seen more adoption of the redhat docker (lowercase d) replacements after Docker (uppercase d) started pushing some of their monetization efforts.
The conclusion that I have to reach is that there are more docker users on Macbooks than I realized.
Last time I tried podman it looked very cool being rootless and daemonless, except that since it's rootless I couldn't create the necessary network interfaces... making it pretty much worthless.
If GP thinks the selling point of podman is that it doesn't have to run as root, but to actually do anything meaningful they have to run it as root after all, why would they use podman?
Many people say that, but don't feel like it's very important.
Having a deamon or not is a technical detail that most people do not care about in my opinion. And it has advantages too, like accessing Docker remotely or from another VM on the same host, or directly from the host which is nice for Docker on Mac or Windows.
After reading this thread, I realize that the 4 GB (average) of my vagrant box images wasn't that bad in retrospect, and that loaded a fully baked OS with packages from apt and just required VirtualBox and its extension pack. (this was not an optimized setup, I was such a noob at the time, just trying to wrangle our developer tools)
The only reason I moved to docker years ago was I wanted a tighter reproducible workflow (ie docker-compose up) for all other developers, turns out vagrant has (at some point in the last 5 years) solved this problem too, with packer (I didn't know about this 5 years ago for whatever reason)
Makes me long for going back to that. At least once you built your base image, it was done and you could just make fast linked images from there. I bet with Alpine you could get a vagrant up in a few minutes tops, then it boots in seconds.
Never liked Ruby as a config language, but doing complex things like setting up shared networks and folders was a breeze comparatively I felt.
Never used vagrant in a production capacity though. Everywhere I ever worked always deployed to bare metal or essentially we bought our services (cloud functions, semi-managed containers etc)
I think there's a disagreement about what Docker-in-Docker actually is. Some people think it's running a docker daemon inside a container, and others think it's accessing the host docker daemon from inside a container by mounting the docker socket as a volume
Sometimes I feel grateful having let all these ephemeral things things pass by without having needed to learn something that was going to stop being "in" so soon.
Containers are going to be there for a while and while the engine running the containers is not always the same, the command line Docker client and the Dockerfile DSL seem to become the standard for Linux containers.
Podman, buildah, and skopeo are insanely awesome tools that let you transition an image from a public registry to your own private deployment in a jiffy. The article doesn't mention it but podman can also dump a pod configuration you can use in kubernetes and open shift/okd so if you're not a yaml wizard you can still get stuff running containerized.
Nope, it's still not there, simplest things work but as soon as you diverge to something more complex it fails.
I tried using it as Docker replacement, but various tools that use docker (using dockerized pip in serverless framework) and complex docker-compose files (dockerized Magento) were broken.
I've also personally struggled a lot with this at $WORK, but some of the new features in RHEL 7.8+ (specifically GA'ing of the fuse-overlayfs graph driver) have actually alleviated a ton of the issues I've seen previously. If you haven't tried that I'd certainly recommend it, YMMV
I just installed my new $WORK docker-compose using podman and podman-compose, and it worked out of the box. To be fair, I failed at it one time before at previous job... So... Coin flip?
Any docker Podman corner cases that people have run into? I like the idea of rootless containers but I can do that ether easily by adding one line to my dockerfile or adding a user flag when I run a docker container. What other advantages am I getting?
Not sure if you count it as a corner case, but podman lacks a docker-compose equivalent - last I checked, a few months ago, there was only a community port project that still needed a lot of work.
The biggest issue lies not in the few lines in /etc/passwd or /etc/group, but rather the highly privileged process with a large attack surface that is the Docker daemon.
Generally, it's been pretty easy to transition for me, but I used to rely on things like the automatic nginx image that you need to mount the docker socket into so it can route traffic to containers when they come up. Since there's no daemon, there wasn't a way to do that. I think people are working on it, but I haven't tried recently.
Work also uses a special docker RUN command that can use the host's ssh keyring to, eg. Install something from a private github repo. Doesn't seem to exist in podman.
I use that nginx image too, and it's the one thing that keeps me using docker for some small single-node setups. For basically everything else, Podman is a better experience. Now that I've discovered Podman can run K8s Pods (and now Deployments) I use it all the time. I don't know how I lived without it!
Does Podman let you get the real SHA-256 digest of the image, pull it to a machine, and then launch the image only if it matches the specified SHA-256 digest?
The podman 'create' command docs [1] do not list a '--digest' argument. I found no example of specifying the digest as part of the image name.
Docker does not support this. You can get an image file and calculate the SHA-256 digest of it. But Docker does not let you say "start a container using this particular SHA-256". That's because the SHA-256 that docker uses internally is just a hash of dockerd's internal metadata about the image. That metadata is different on every machine. I felt extremely disappointed when I discovered this. And I lost a lot of respect for the Docker developers.
If you want to deploy based on image hash with Docker, you must add your own verification step before creating each container instance. Terraform's docker plugin does not perform this step. And the local dockerd will not check the SHA-256 on reboot. If you just follow Docker's "best practices" and use the provided tools, all of these sources will needlessly have root on everything you deploy:
- Your docker image repository
- Any machines with credentials to push to your docker image repository (CI system, engineer machines, anyone with access to engineer machine backups)
- Anyone with permission to push any public image that you use. The way most CI systems are configured, all jobs get the same credentials. So this includes your build and test jobs. That fancy linter Docker image you're using? Anyone who can push to its Dockerhub account can replace the image version you selected with one that roots all of your systems.
- Dockerhub itself
Why would RedHat switch to a Docker replacement without this crucial security feature?
Am I missing some important point that makes all of this ok?
Assuming that it's Red Hat's fault that there is no el8 package for Docker, I think Red Hat is really shooting themselves in the foot by not supporting Docker in RHEL8. (You can still install the el7 version of Docker on el8 by performing an enable/disable/enable/disable/... module dance, but... why isn't there just an el8 version?)
Yes, but not Docker. (Also Podman is developed by Red Hat.) I think they have rather high hopes there, and are more likely to just shoot themselves in the foot by being one of a small number of distros that don't have Docker. In fact, rebuilders of RHEL (e.g. Oracle Linux) should maybe consider explicitly supporting Docker. "We are 100% compatible with RHEL but additionally also support Docker" definitely makes for better marketing copy than just "We are 100% compatible with RHEL".
I am still thinking whether to include Docker or Podman in my book[0]? Since I am leaning towards RH tech (like SELinux), I think I will go with Podman. I think it's the future on Fedora-based systems.
Lack of a drop-in docker-compose replacement is what stopped me last time I tried podman. podman-compose looked decent but it lacked a bunch of basic features and didn't work smoothly last time I tried it.
Is the Docker daemon actually a huge security hole?
I see people championing Podman because it's daemonless but is it actually beneficial or is it championed because it's Red Hat and a case of security check boxing?
It's a root daemon which can be made to run arbitrary commands by anyone using its interface, so yes it is a huge, huge security burden. You can't just brush that off as a superficial problem.
Given that the most common case is running docker in server environments in VMs and the sysadmins are root, is this a real issue? Can you tell me an environment where a multi-tenant system runs docker?
Yes, it[1] is[2] a[3] real[4] issue[5] which leads to privilege escalation bugs. It also doesn't help that most containers that Docker is responsible for managing is supposed to be unprivileged, and Docker itself is commonly used as a component for a multi-tenant container runtime.
Do you mean in that adding a user to the docker group is effectively handing them root? Because if you don't do that, then running docker requires root/sudo, which means that it should be exactly as secure as anything else
I'm an enduser and I expect things to "just work". That seems to match a lot with Red Hat's Linux distros (i.e. RHEL): "Just use it, it's a defacto standard so most third party software is tested on it, don't worry about underlying components as Red Hat has made sane choices for you."
That's the theory (and Red Hat's marketing). I wish I could just be using CentOS/RHEL and not worry about the choices Red Hat makes. But the shit is piling up (sorry for my harsh words, no offence intended).
So, let's containerise something simple on top of the "centos" container image, let's say postfix. Oh, postfix on centos needs systemd for logging. But systemd in a container is a nightmare. You can get systemd to work in a container if the host system is using containerd and you pass certain special files from the host to the container. So just use Podman instead of Docker, right? Podman has functionality to make systemd in a container work. OK, I can run my container on my fat workstation (a Centos system with Podman). But is it still portable? Can I just run it in Github Actions, AWS ECS, a Windows machine with Docker Desktop? Nope. It's not portable.
Let's put the systemd rant aside (I really like the systemd CLI UX, but the architecture (dbus, etc) seems to only serve one usecase properly (Desktop systems) and seems to be heavily wrong for the container usecase).
Let's rant about the architectural choices Red Hat is making for containers.
Before Red Hat started to get involved there was the open source, (now) cross-platform containerd that a lot of tools are building on top. It consumes the low level runtime runc and both provides an API for Kubernetes (CRI) and additional things. It is highly pluggable and therefore the "only container runtime you'll ever need". And it's purely community governed (CNCF).
What is Red Hat doing? Are they building their container tools on top of containerd? Nope. They create their own low level container runtime (crun), their own mid level container runtime (cri-o). But cri-o only covers what's needed for Kubernetes. So to be able to build container images, etc (the "working with containers on a single machine" usecase) additional tools are needed (podman, buildah, skopeo) and they have to implement the missing functionality themselves. So Red Hat's Open Shift (Kubernetes distribution) builds on an entirely different stack than Podman.
Fragmentation everywhere. Was that really necessary?
Why should I care as an enduser? Well, I can inspect and debug all tools that use containerd the same way (i.e. using the "ctr" tool). I can relatively easily even write my own tools that use containerd (via it's grpc api) and can access the resources (containers, images, etc) that these tools are managing to solve my special custom needs.
For the Red Hat stuff everything is different. I could work with cri-o using crictl and write custom tools using cri interface. For Podman I would need to use Podman's API. (Which has annoyingly entirely changed between v1 and v2).
ok, enough rant (I could go on with how hard it is to get an up to date version of podman on a supported RHEL system even with app streams, that UBI images don't provide podman/buildah/skopeo, etc etc etc).
Please don't mistake me: Podman, cri-o, crun, open shift are all amazing technologies. And it's great that they are mostly open source via upstream projects. But I wish this whole fragmentation hadn't happened. For me as an enduser it's nightmare.
Disclaimer: I work for Red Hat, but I don't code or design any of the things you've mentioned here. Opinions are my own.
I think I just gave you your first ever upvote on HN!
Thanks for this post, it's great discussion.
Regarding Postfix/systemd issues, you should take a look at ubi-init image. It has a minimal systemd in it to address problems like that[1][2].
Regarding crun, have you looked at the "Why another implementation" in the crun README.md by chance?[3] Or have you read the blog post about it that goes into a lot more detail?[4]
I totally agree with the dislike toward fragmentation. I myself have been very critical of NIH and fragmentation when it happens. I do think there's an interesting case to be made for crun though.
thanks for the response and the links! I will certainly have a look.
(And again sorry for my rather harsh words - I have a lot of respect for all the hard work everyone at Red Hat is doing. I can only imagine that it's much harder to make technical decisions and build tools like Podman than it looks like. Thank you for not taking my rant as an offence! It's just my brutally raw user experience).
containerd is a daemon, which actually came out after CRI-O. But anyways the hole idea is Podman and Buildah was to get away from using root running daemons to launch containers. The important thing about containers is that we launch we all follow the standards. Podman, Docker, CRI-O, Buildah, Containerd all use the same OCI based images available at container registries, and all launch them the same time via OCI Runtimes. These runtimes can be runc, crun, kata, krun, gVisor ...
There is no fragmentation on OCI, which is important. Think of these as been web content, you have lots of choice on viewing and using web content, firefox, chrome, safari, IE, curl, linx, wget... All of these tools (Podman, Buildah, Skopeo, CRI-O) that can interact with the OCI content and all have different strengths.
Please do not add FUD about systemd.
None of the new container engines require systemd, they all can run and support all images at Docker.io. The new container engines support systemd better then Docker did, but they do NOT require it.
I don't think he's FUDding about systemd. I may have misinterpreted, but I think he is saying that his application (postfix in the example he gave) requires systemd as a dependency (for logging in this example). I don't think he is saying that podman, et al require systemd.
I would respectfully offer as well that from your perspective there's no fragmentation (and I think you make a great point about that), but from the perspective of a typical developer or sys-admin, there is absolutely fragmentation. A lot of devs/sysadmins don't only work with Red Hat, they have other linuxes to deal with as well. To them, it used to be that `docker` and everything about it was the standard. Whether you wanted to develop on Fedora or Ubuntu, it didn't matter, you just used `docker`. When you went to prod, it was just docker. Everyone on the internet writing blog posts and tutorials were using `docker`. There really wasn't anything else that was popular (although I was personally rooting for rkt and systemd-nspawn but those didn't really take off ;-)
Nowadays, there is "fragmentation" because depending on you production platform, your OS of choice, architect whims, or whatever, at any given time you may be dealing with docker, or podman, or CRI-O etc.
That said, my personal perspective is closer to yours. I agree with your arguments, and I'm a big fan of Podman et al and I don't see it as fragmentation. When I put on the perspective of an average dev/sysadmin (which I work closely with every day), some of the terms (like "fragmentation" change meaning a bit).
The point that I wanted to make about systemd is that in practice I'm again and again running into systemd vs. containers problems. Systemd isn't really intended to run in containers. There are workarounds (such as sharing special files from the host with the container or using Podman that has implemented similar workarounds for it). The reason is that on modern "fat" Linux distributions (like RHEL/CentOS or Ubuntu) more and more things depend on systemd. Postfix is just a simple example. Because of this clash the original promise of containers (package any user mode stuff into a container and run in anywhere) often feels broken. This isn't Podmans fault (Podman is doing what it can to ease the pain, but it only works when you use Podman). A container image that contains systemd needs special treating (not possible on any host or platform that can run containers) and isn't really a portable container anymore.
These kind of issues tend to let me avoid using CentOS/UBI as container base image and instead go with i.e. Alpine that doesn't have the things I like about EL (LTS, etc) but also never has any mixups with systemd. (It's a bit sad having to go that road when your workplace actually pays for EL support).
I wouldn't have any suggestion how to easily fix this though. Systemd will never fully run in a container I suppose (it's too much linked to non userspace) nor will all container tools and platforms support workarounds for it.
Aside from systemd here's another example where I think Podman UX could be better. Running containers in containers is an important usecase, even if it's insecure. Docker has provided workarounds for this early on. Yes, horrible in terms of security. But good enough for CI/CD runners. Either you can pass the Docker sock file from host to guest. Or in a --privileged container you can run a Docker engine and connect to it from another Docker container. It's well documented and there is a ready "docker:dind" (docker in docker) container image.
My #1 hope with Podman rootless containers was that I could just run "podman run ..." inside a podman container. But it turns out it's (sill) not that easy, even with "--storage-driver=vfs".
What's the recommended approach for running containers in containers with a version of Podman that is available today in the stable version of RHEL/CentOS? I lack to find docs about it. I assume you can do some unprivileged foo with Podman, too ... but how?
(I know latest Podman2 is providing a largely Docker compatible sock file, but I understand its early? Don't misunderstand me, this is great, will use it once it lands in RHEL. But until then ... Docker for this?).
and yes, people have proven that it's possible to build something like podman or buildah on top of containerd / buildkit! No need to reinvent the wheel!
RedHat seems to be pushing a standard ecosystem for Linux: systemd, Wayland, SELinux, GNOME, and now maybe podman. I've been on Linux for a while; it's a welcome change from all the fragmentation I'm used to. Whereas others try to work around the kernel and implement their own things in parallel (see: Canonical's AppArmor, LXC, OpenZFS), RedHat just goes with what Linux already has like SELinux/cgroups v2/btrfs, which I think is more likely to last and just feels better. If RedHat goes away, I'm fine since I'm ultimately only relying on Linux features. If Canonical goes away, then I'd have to switch to a different stack. That's probably why government, enterprises, Amazon, etc. still prefer RedHat.