> My problem is solely with the Nix community. The amount of jargon-loving and argumentative "you shouldn't attempt what you are attempting in the first place" or "you are holding it wrong" gaslighting is really something :D
> I think the point where you fail to grasp the problem is: Nobody is claiming that Nix can't do a thing. My complaint is: There is no way to find _the_ way to do the thing. And there seem to be multiple _incompatible_ ways to achieve any given state. Without any way to distinguish those (let's call them) camps for a new person. In this way it's everything Unix but dialled to eleven! You don't need to argue (for me) that anything _can_ be done. I'm getting that from "Turing complete". My problem is only the practice: How do I get there?
I honestly have relatively little experience with the majority of the Nix community, so I can't speak too much to it. I did not personally experience too many problems.
The frustrating thing to me about this thread is that it definitely wasn't in this vein. I know I'm harping on this a lot, but it matters.
What the blog post says is "we can't split the image into multiple layers with Nix". And again, they're doing their own OCI building, so this actually has little to do with Nixpkgs and is much more conceptual in nature. If you build a Nix closure, what you wind up with is a number of store paths. For example, for bash, you might have the following store paths:
So how do you split these into layers? Well, however you want. You can place them into individual layers, or you can group them based on dependencies to make more "logical" groupings and get less layers, or basically any number of things. There's plenty of metadata on what depends on what, so it's really up to you to decide how to do that.
I was not talking to end users when I said this, I was talking to developers building on Nix that almost definitely knew this. So it's unclear based on what they said why they couldn't.
> edit: I think my immediate problem is: How do I get the series of "commands" that a nix derivation will invoke? Kind of like "what's the receipt, that is being executed?" That's missing for a sensible repl-loop for me. I'm sure I'll be getting there. The capability that Nix offers is extremely appealing. But at the same time the documentation and community are unhelpful to the extreme
I see.
Well, for starters, you are trying to start at the very low-level bottom of the stack. I understand this because it is the approach I prefer to take when learning new technology, but I'm telling you right now this is a hard road to go down with Nix, I've been down it. The documentation is going to continue to feel unhelpful because you will find plenty of documentation that is happy to take you down the dark paths of how things actually work, but that doesn't make them any easier to navigate. It's much easier to start by asking "how do I accomplish this end-goal" than to ask "what actually is a derivation".
With that out of the way... to understand what's going on with builds, you really genuinely do need to know what a derivation is, because it matters. Almost nobody calls the `derivation` function directly in Nix, instead using Nixpkgs mkDerivation tool at the bare minimum, but at the very bottom, there is only one function that matters:
At the bottom, in order to actually build something, a derivation has to be evaluated and realized. Internally, instantiating a derivation will cause it to be built. It is built by taking the parameters passed to derivation and serializing them to an internal format, then "instantiating" it, causing it to be built inside of the sandbox. The internal derivation format looks like this:
So what does `nix build nixpkgs#bash` do? Well, aside from the sandbox setup, on my particular machine, it would run... /nix/store/razasrvdg7ckplfmvdxv4ia3wbayr94s-bootstrap-tools/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh. So, basically, a bash script. What does that bash script look like? Here's the entire contents of default-builder:
genericBuild
This is in the heart of the Nixpkgs "stdenv" machinery, and now we're outside of what Nix itself does and into what Nixpkgs does. Literally all Nix does is handle the book-keeping: sandboxing, execing the builder, handling the Nix store; for the actual build, it is just execing the builder, and in most cases, it will wind up being handled by a bash script. You could have whatever builder you want here, of course, and it needn't be bash or bash-based. (Note that we're calling bash to build bash - funny choice for me to pick, but you can see how it gets the copy of bash to build bash with from a special derivation called bootstrap-tools. Gotta break the cycle somehow, and Nix doesn't have as impressive of a bootstrap seed as Guix.)
So, for this build, it just calls genericBuild. What does that do? Well, you can keep chasing down the scripting here, but ultimately it calls all of the build phases, e.g. unpackPhase, patchPhase, configurePhase, buildPhase, checkPhase, etc. which are all just bash scripts inside of environment variables. At the end of the day, it does roughly what you would do to build bash, plus some patches. The nixpkgs stdenv automation helps handle many special Nix things, but it's not actually horribly complicated.
By the way, when you see this sort of thing in a Nixpkgs derivation:
patchPhase = '''
...
''';
It is in fact just setting up environment variables. You can see them in `nix derivation show`, even. They get called just by substituting them directly into a command, not any different than:
test="echo Hello world" bash -c "$test"
Basically, all of the attributes in the mkDerivation call become environment variables, including src, and all of the magic that gives them meaning is just in the stdenv default builder. (This is roughly true, anyway. I think there is some move to make it "stricter" but the default way it works is still like this roughly.)
So how can you debug what actually happens when building a derivation? Unfortunately it's not terribly easy. You can view the logs with `nix log`, but it will only show one if it actually built it, not if it substituted it from Hydra cache for example. Also, the default behavior of the builder doesn't output that much diagnostics, though you can add NIX_DEBUG = 7 in a derivation and it will output tons of information about what is being executed.
The way derivations depend on other derivations is something like this:
- During build-time, when you try to coerce the result of the derivation function to a string, what gets returned is the realized store path. So if I try to do `${pkg.bash}/bin/bash` inside of Nix, it might return `/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin/bash`. In this way, it is natural to literally just depend on something implicitly by realizing it as needed.
- As far as I know, for runtime dependencies, it is detected by literally scanning the $out directory after a derivation is built and looking for the Nix store paths of other derivations. Merely realizing a dependency during build-time won't make it a runtime dependency.
It's more complicated than this, but that's the broad strokes. So when you build bash, it will implicitly depend on all kinds of build tools like GCC and make, but the resulting derivation will only depend on things like libgcc that are actually referenced in the final derivation.
Honestly, it is a bit of a black box. It is not easy to navigate. Trying to learn Nix from the ground up first will be very hard. I wouldn't ever tell someone not to try, but I will definitely say it won't be easy.
> I think the point where you fail to grasp the problem is: Nobody is claiming that Nix can't do a thing. My complaint is: There is no way to find _the_ way to do the thing. And there seem to be multiple _incompatible_ ways to achieve any given state. Without any way to distinguish those (let's call them) camps for a new person. In this way it's everything Unix but dialled to eleven! You don't need to argue (for me) that anything _can_ be done. I'm getting that from "Turing complete". My problem is only the practice: How do I get there?
I honestly have relatively little experience with the majority of the Nix community, so I can't speak too much to it. I did not personally experience too many problems.
The frustrating thing to me about this thread is that it definitely wasn't in this vein. I know I'm harping on this a lot, but it matters.
What the blog post says is "we can't split the image into multiple layers with Nix". And again, they're doing their own OCI building, so this actually has little to do with Nixpkgs and is much more conceptual in nature. If you build a Nix closure, what you wind up with is a number of store paths. For example, for bash, you might have the following store paths:
So how do you split these into layers? Well, however you want. You can place them into individual layers, or you can group them based on dependencies to make more "logical" groupings and get less layers, or basically any number of things. There's plenty of metadata on what depends on what, so it's really up to you to decide how to do that.I was not talking to end users when I said this, I was talking to developers building on Nix that almost definitely knew this. So it's unclear based on what they said why they couldn't.
> edit: I think my immediate problem is: How do I get the series of "commands" that a nix derivation will invoke? Kind of like "what's the receipt, that is being executed?" That's missing for a sensible repl-loop for me. I'm sure I'll be getting there. The capability that Nix offers is extremely appealing. But at the same time the documentation and community are unhelpful to the extreme
I see.
Well, for starters, you are trying to start at the very low-level bottom of the stack. I understand this because it is the approach I prefer to take when learning new technology, but I'm telling you right now this is a hard road to go down with Nix, I've been down it. The documentation is going to continue to feel unhelpful because you will find plenty of documentation that is happy to take you down the dark paths of how things actually work, but that doesn't make them any easier to navigate. It's much easier to start by asking "how do I accomplish this end-goal" than to ask "what actually is a derivation".
With that out of the way... to understand what's going on with builds, you really genuinely do need to know what a derivation is, because it matters. Almost nobody calls the `derivation` function directly in Nix, instead using Nixpkgs mkDerivation tool at the bare minimum, but at the very bottom, there is only one function that matters:
https://nix.dev/manual/nix/2.22/language/derivations
At the bottom, in order to actually build something, a derivation has to be evaluated and realized. Internally, instantiating a derivation will cause it to be built. It is built by taking the parameters passed to derivation and serializing them to an internal format, then "instantiating" it, causing it to be built inside of the sandbox. The internal derivation format looks like this:
Basically a lot of arrays and tuples and whatnot. Not very easy to read. `nix derivation show` can give us better human-readable output: So what does `nix build nixpkgs#bash` do? Well, aside from the sandbox setup, on my particular machine, it would run... /nix/store/razasrvdg7ckplfmvdxv4ia3wbayr94s-bootstrap-tools/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh. So, basically, a bash script. What does that bash script look like? Here's the entire contents of default-builder: This is in the heart of the Nixpkgs "stdenv" machinery, and now we're outside of what Nix itself does and into what Nixpkgs does. Literally all Nix does is handle the book-keeping: sandboxing, execing the builder, handling the Nix store; for the actual build, it is just execing the builder, and in most cases, it will wind up being handled by a bash script. You could have whatever builder you want here, of course, and it needn't be bash or bash-based. (Note that we're calling bash to build bash - funny choice for me to pick, but you can see how it gets the copy of bash to build bash with from a special derivation called bootstrap-tools. Gotta break the cycle somehow, and Nix doesn't have as impressive of a bootstrap seed as Guix.)So, for this build, it just calls genericBuild. What does that do? Well, you can keep chasing down the scripting here, but ultimately it calls all of the build phases, e.g. unpackPhase, patchPhase, configurePhase, buildPhase, checkPhase, etc. which are all just bash scripts inside of environment variables. At the end of the day, it does roughly what you would do to build bash, plus some patches. The nixpkgs stdenv automation helps handle many special Nix things, but it's not actually horribly complicated.
By the way, when you see this sort of thing in a Nixpkgs derivation:
It is in fact just setting up environment variables. You can see them in `nix derivation show`, even. They get called just by substituting them directly into a command, not any different than: Basically, all of the attributes in the mkDerivation call become environment variables, including src, and all of the magic that gives them meaning is just in the stdenv default builder. (This is roughly true, anyway. I think there is some move to make it "stricter" but the default way it works is still like this roughly.)So how can you debug what actually happens when building a derivation? Unfortunately it's not terribly easy. You can view the logs with `nix log`, but it will only show one if it actually built it, not if it substituted it from Hydra cache for example. Also, the default behavior of the builder doesn't output that much diagnostics, though you can add NIX_DEBUG = 7 in a derivation and it will output tons of information about what is being executed.
The way derivations depend on other derivations is something like this:
- During build-time, when you try to coerce the result of the derivation function to a string, what gets returned is the realized store path. So if I try to do `${pkg.bash}/bin/bash` inside of Nix, it might return `/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin/bash`. In this way, it is natural to literally just depend on something implicitly by realizing it as needed.
- As far as I know, for runtime dependencies, it is detected by literally scanning the $out directory after a derivation is built and looking for the Nix store paths of other derivations. Merely realizing a dependency during build-time won't make it a runtime dependency.
It's more complicated than this, but that's the broad strokes. So when you build bash, it will implicitly depend on all kinds of build tools like GCC and make, but the resulting derivation will only depend on things like libgcc that are actually referenced in the final derivation.
Honestly, it is a bit of a black box. It is not easy to navigate. Trying to learn Nix from the ground up first will be very hard. I wouldn't ever tell someone not to try, but I will definitely say it won't be easy.