Millions of transactions per second is not the point of using docker.
Docker is about running multiple things on the same server. If you have 1 server and you have to run 1 app on it, you don't need a Docker container.
Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it. What is hard is that running multiple apps, you run into issues with dependencies. Ideally you run each app on different server or separate OS then you don't have that problem, but that is costly and if one app sits idle you waste money you spent on the server.
I don't know why people think about it the other way that you get 1 app and run it distributed is "peak containers". Peak containers is about having common infra so you make the most of your server because you don't care what app is running in there, your server does not need to have any special config for your special app. You don't get bogged down that you cannot install anything else on that server because of dependency hell.
Well from the application point of view bonus is that if you dockerize your app you don't care about the servers it is running on. You just ship it and it will run on someones server with other apps that you don't know about and don't care about and their dependencies will not break your app.
> Docker is about running multiple things on the same server
Or about running a single thing on multiple servers. Or providing a temporary runtime with language/lib/software X preinstalled. Or to facilitate easier testing.
In the end, many people use containers for many different reasons. There isn't a single "This is what docker is for", although people tend to shove containers into places that might not be 100% suitable for containers.
> What is hard is that running multiple apps, you run into issues with dependencies.
That's exactly the point they were making in this article. Also even with 1 server and 1 app people still use Docker because the pain of setting up a Linux machine with all of the dependencies for your average Linux software is so great, and Docker simplifies that.
The single static binary is such a great feature of Go. I wish more languages would copy it.
> Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it.
There are so many options other than Docker as of today. These provide very different tradeoffs. Somehow we got to the point that people think that there are only one solution (Docker) to run multiple applications on a single server.
Docker has tooling for building images and is lighter than VM.
Devs can make their docker containers quite easily. With VM you have to update operating system and other stuff. With Docker as a Developer you are concerned only about your direct dependencies.
I run my systems in VMs and it is quite a lot of work to update all unrelated stuff. Well related stuff is also a lot of work but that is price I am willing to pay for.
I use Nix for building projects and managing my laptops (NixOS at home, nix-darwin on macOS at work). This solves the 'dependency hell' in a much cleaner, more efficient and IMHO simpler way than docker.
For those who don't know, Nix doesn't install anything system-wide. Everything gets prefixed with a hash of its dependencies (e.g. /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1/bin/git). Since nothing is available system-wide, program A can only use program B by specifying its whole path (including the hash); this affects the hash of program A, and so on for anything that refers to it. This is a Merkle tree, giving the provenance of every dependency, all the way down to the 'bootstrap' tools (e.g. a prebuilt GCC that was used to build the GCC that was used to build the tar that was used to extract the bash source ....). Here's a snippet of the dependencies used to build the above git command:
Here's a snippet of the above git command's runtime dependencies (i.e. everything it references which, if it were our app, we'd need to include in its container):
All of these builds are performed in a sandbox, with no access to the network or global filesystem (e.g. no $HOME, /usr/bin, etc.), every file's timestamps are set to 1970-01-01T00:00:01 and the /nix/store filesystem is read-only.
The Guix project takes this a bit further, with its `guix pack` command turning paths like this into a runnable docker image (amongst other options).
Compare this to the way docker is used: we download and run some large binary blob ('image'), containing who-knows-what. These images are usually bloated with all sorts os unneeded cruft (due to the way docker builds images 'from the inside'); often a whole OS, package management tools, or at the very least a shell like busybox, all of which makes scaling harder and increases our attack surface. The build steps for those images are often insecure and unreproducible too, e.g. running commands like `apt-get install -y foo` or `pip install bar`, whose behaviour changes over time dependending on the contents of third-party servers.
I really want to start using containers for their security and deployment advantages, but so much of their tooling and approach seems completely backwards to me :(
(Note that it's possible to build docker images using Nix, but this requires running commands like 'docker load' which I can't get working on my Mac, and deploying to AWS ECS seems to require 'image registries' rather than just pointing to a file on S3. Urgh.)
Totally unrelated question: Do Nix packages typically come with proper license & copyright notes (and source code, if necessary) and is there an easy way to extract them for the entire dependency tree?
The reason I'm asking is that, as our team is getting closer to ship our $PRODUCT, we started worrying about the licenses of all the third-party software components we use. Needless to say, we're using some Docker image as base and now need to figure out what software it contains exactly…
Yes, packages in Nixpkgs (the official repository of Nix packages) have mandatory copyright or license information.
This can also be accessed programmatically, if you intend to scan all your dependencies:
A large majority of the packages is built from source and so you can easily inspect it. For example, running `nix-build
'<nixpkgs>' -A vim.src` will fetch the source tarball and link it in the current directory.
In addition to what rnhmjoj says, Nix can also check the license of packages as it's evaluating them (i.e. before fetching/building). The most obvious place this appears is the `allowUnfree` option, which defaults to `false` and refuses to evaluate proprietary packages. Details on how to whitelist/blacklist specific licenses, and how to define custom functions to decide this, are given at https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree
Nix is completely based around source code, similar to Gentoo's emerge, BSD ports, etc. (in contrast to those based around binaries, like dpkg, RPM, etc.).
More precisely, Nix works with "derivations", which have outputs (the paths in /nix/store which they define), inputs (the outputs of other derivations), and a "builder" (this is usually bash, with a build script given as argument).
The files above which end in `.drv` define derivations. Fun fact: Nix doesn't care how these files are made; we can write our own tools to create .drv files, which is exactly what Guix and hnix do :)
Notice that the runtime dependencies above don't contain any .drv files. That's because at runtime we only need (some of) the outputs of (some of) the derivations.
Also note that the build dependencies contain lots of .drv files, since the build instructions are just as much a part of the 'source' as anything else.
Some of those derivations have names like `.tar.gz.drv` and `.xz.drv`; those define how to fetch a particular archive, e.g. containing a URL, maybe a bunch of mirrors, a SHA256 hash to compare it against, etc. Internally, Nix doesn't distinguish between such "sources" and anything else; it's all just derivations.
Since Nix builds tend to be reproducible (although that's not guaranteed), it's able to use a clever optimisation trick: packages (i.e. the outputs of derivations) are completely specified by their hash, so we can copy them from someone else (if they already have it), rather than building them ourselves, and the result should be the same. Similar to how we can checkout a particular git commit from someone who already has it, rather than having to recreate the tree ourselves by make all the same edits in the same order. (It's not exactly the same, since we need to trust that person isn't sending us something malicious).
Hence Nix can use a (trusted) binary cache: this is just a folder full of derivation outputs, which we can query to see if our desired output has already been built, before we bother doing it ourselves.
Binary caches are the reason I can do `nix-shell -p git` and have a git binary ready to go in a couple of seconds, despite that binary being defined in terms of those massive source-based dependency graphs.
Small docker image: look into Alpine which comes in at ~5MB. You are basically starting from nothing.
Versioning: both apt and pip allow you to specify a version. I dont think this is a docker-specific issue. I'm sure there are docker images with old packages on them. Or i suppose you could just run a tool that downloads specific versions of binaries (like nix) in your Dockerfile.
Absolutely. I would ultimately like a container image (docker format or otherwise), containing precisely the files that are needed by the intended service, and nothing else. In particular, the container should not contain unwanted cruft like wget, pip, apt, vi, cp, etc.; and it certainly shouldn't contain a shell (bash/dash/busybox/etc.).
The way docker runs containers is fine (albeit overly-complicated for my tastes, e.g. "loading an image from a registry", compared to something like `kvm -hda myImage.img`)
The way docker's tools build those images is bad. Hence why Nix, Guix, etc. are better for that.
> Small docker image: look into Alpine which comes in at ~5MB. You are basically starting from nothing.
Nope, that is starting from an entire OS. Even a busybox-only image is made dangerous by the existence of a shell, since we have to worry about exploits letting attackers 'shell out' (and from there, inspecting and altering the application binary, not to mention trying to jailbreak or remote-out of the shell to do damage elsewhere, etc.)
> Versioning: both apt and pip allow you to specify a version.
I don't particularly care about versions; I want to specify the SHA hash, so I know if something has changed on the third party server (pypi, ubuntu, or whatever); or whether my connection has been re-routed maliciously; or the download was interrupted/corrupted; or some distant config/setting has changed the resolver's behaviour; and so on.
I also want some way to enforce this, so that no file can exist in the container without being verified against an explicit SHA: either directly, or extracted from an archive with a specified SHA, or checked-out of a particular git commit (which is itself a SHA), or built in a deterministic/reproducible way from components with known SHAs (e.g. a tarball, GCC binary and Makefile, all with known SHAs), etc. I want a fatal error if I forget a SHA.
A system like that would be tolerable, but still an uncomfortable step backwards.
Verifying SHAs is good, but we'd still be running random binaries fetched from a third-party. Unlike apt, pip, etc. Nix gives me the full provenance of everything: I can, at a glance, see whether a particular patch was applied to the GCC that compiled the bash that was used to build my binary; or whatever (admittedly, this is more useful for auditing things like OpenSSL, rather than trusting-trust attacks on GCC).
I can also, with a single command, recreate any of those build environments in a shell, e.g. if I need to test the presence of a vulnerability or bug. As a by-product, I can also use those for development, jumping in to any step along the provenance chain to edit files, run tests, etc.
I can also choose whether or not I want to trust the builder/provider of those binaries, or whether I want to build them myself, and I can choose at what level I want to place that trust; e.g. maybe I trust their GCC, but I want to build the JDK myself. As a by-product, I can also switch out or alter any of those steps, e.g. enabling a GCC flag for a certain build, completely replacing the JDK with another, etc. The Nix language, and the architecture of the Nixpkgs repository, makes this very easy; and Nix itself takes care of rebuilding everything downstream of such changes.
To understand this, I like to think of Nix more like an alternative to Make, rather than apt/dpkg/pip/etc. Those tools are band-aids, created because Make doesn't scale or compose very well. Similarly, Dockerfiles/Chef/Puppet/Ansible/etc. are band-aids created because apt/dpkg/pip/etc. don't scale or compose well. Nix does compose and scale well, so it can be used across that whole spectrum of 'get result FOO by running commands BAR' (we can take it even further with projects like Disnix and NixOps which perform orchestration, but I've personally never used them).
> Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it. What is hard is that running multiple apps, you run into issues with dependencies. Ideally you run each app on different server or separate OS then you don't have that problem, but that is costly and if one app sits idle you waste money you spent on the server.
I mean yeah, it was sortof a problem on late 90s x86 kit. But its not been a real problem for a good 20 years. virtualisation solved that issue. Modern processors (well not even modern ones, any xeon from the last 10 years) will have virtualisation optimization in them.
Docker allows you to not be as disciplined when it comes to your build environment. This means that you can let developers ship any old shit. This will give you a short term productivity boost, and longterm tech debt.
> your server does not need to have any special config for your special app
yeahna, thats not really true. You still need to manage secrets (docker has no real security bits built in, barring cgroups) you still have to manage volumes, and you still have to manage connectivity. Not to mention resource discovery.
The biggest lie that K8s and docker told us is that you didn't need a coherent plan for your infra. To be efficient, you need to make your apps work on the same dependencies, same platform, and re-use as much as possible.
Docker is about running multiple things on the same server. If you have 1 server and you have to run 1 app on it, you don't need a Docker container.
Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it. What is hard is that running multiple apps, you run into issues with dependencies. Ideally you run each app on different server or separate OS then you don't have that problem, but that is costly and if one app sits idle you waste money you spent on the server.
I don't know why people think about it the other way that you get 1 app and run it distributed is "peak containers". Peak containers is about having common infra so you make the most of your server because you don't care what app is running in there, your server does not need to have any special config for your special app. You don't get bogged down that you cannot install anything else on that server because of dependency hell.
Well from the application point of view bonus is that if you dockerize your app you don't care about the servers it is running on. You just ship it and it will run on someones server with other apps that you don't know about and don't care about and their dependencies will not break your app.