Hacker News new | past | comments | ask | show | jobs | submit login
NixOps – Declarative cloud provisioning and deployment with NixOS (nixos.org)
248 points by mrkgnao on Sept 17, 2017 | hide | past | favorite | 66 comments



In view of some replies downthread: yes, application-level management is entirely possible with Nix(OS).

Currently, this works best for Haskell, for which there is a sizeable amount (relatively speaking) of Nix tooling available. This is largely because a lot of Nix(OS) users also use Haskell, perhaps because lend themselves to very similar modes of "declarative" thinking.

This is a thorough, excellent guide on how to develop Haskell projects with Nix, including managing non-Haskell dependencies (zlib, SDL, ncurses, and so on):

https://github.com/Gabriel439/haskell-nix/

I only started using NixOS about a month ago, and it's fantastic. The best part is that binary caches are available for almost everything, so the hitherto-painful step where one compiles Haskell dependencies locally has been completely eliminated. The best way I can describe how I work now is "Python virtualenvs, but for everything at once and just so much better".

PS. Gabriel Gonzalez is, in general, someone who makes great documenation. Check out the tutorial for his stream-processing library Pipes: the ASCII art alone is awesome :)

https://hackage.haskell.org/package/pipes-4.3.4/docs/Pipes-T...


We use NixOS in production in our product, for our cloud services, and for some development environments.

We build everything with nix from sbt projects, go projects, haskell projects, purescript projects, docker images, and npm projects.

It's an incredible tool.

NixOps is also a good tool but there are a lot of issues with it we haven't had time to contribute fixes for. We also wanted it to be lighter weight and to decouple system specification from system deployment. So I wrote a tool that handles performing the same system deployment operations that NixOps does for you and you can simply pipe the nix path result from nix-build to it. We use terraform to specify and manage the cloud resources.


Are you able to share your tool?


We will definitely be publishing it. Time is very short and the backlog of projects to publish is growing though so it might take time to polish and write a blogpost and tutorial.

I work for Awake Security so keep an eye out on the company's GitHub org as we've been publicly releasing a lot of projects even though they aren't completely polished or announced yet. I'll see if I can do this for our deploy tool soon.


Nice, I'd love to see it released.


Gabriel also has a great blog http://www.haskellforall.com/

Basically the kind of person I would love to buy a beer any time because he does so much great work.

on topic: The binary caches are great. For work I'm currently evaluating https://github.com/typeable/stackage2nix and https://github.com/input-output-hk/stack2nix to generate nix expressions from our haskell stack projects. And then push the build artifacts to a shared cache. Such that nobody has to ever compile a dependency twice. Because nothing sucks more than starting your day watching the compiler churn away at compiling after version bumps, to then also see a collegue go to the same pain all over again.


I love Nix concepts and all it's subprojects (NixOS, NixOps, Hydra, Disnix).

However, i've found NixPkgs to be a bit messy. It's too big, targets too many platforms and it's too disorganised. But to be completely fair, I think it's getting a lot of good refactorings lately.


Yeah it's basically just growing pains.

For large enterprise use, Nix is going to need to mature to a point where it's possible for enterprises to host a Nixpkgs cache locally, using a binary repository manager like Artifactory or Nexus, both for sheer cost (to save on bandwidth of thousands of servers downloading updates from the Internet) as well as risk reduction (in case the public binary caches are unavailable at critical moments). If this is doable today, it's not really documented.

I'm sure the project will get there... eventually.


You can install nix-serve on any machine to share its binary cache (including patched applications). Then add all of the enabled machines to every machine's nix.binaryCaches in order of preference. If they don't have what you need, it will fall back to the upstream server. You'll want to make sure they are upgraded in an appropriate order to minimize the need to leave your network for updates. It's true this isn't well documented, but I figured it out in about a day and was surprised at how simple it turned out to be.

You can even go further and set up your own build server to service all the architectures you need to support.


I found Guix a lot more organised. Most package code is very simple, and the toolchain is also quite elegant. Hopefully the latter will be addressed by the new nix interface.


Is GuixSD really usable in production? Its focus on using a Linux-libre kernel instead of mainline is worrisome regarding real-world hardware support.


Switching to mainline is almost trivial, just override plain linux-libre kernel and redefine the source.

GuixSD is getting production-ready IMHO. See for example Guix getting adopted by some HPCs:

https://guix-hpc.bordeaux.inria.fr/


That is already very easy:

https://github.com/nh2/nix-binary-cache-proxy

Allows you to set up (using NixOps!) a caching proxy for the nixos binary cache in your local network.

It can also serve your own custom packages if you sign it with your own package signing key and tell your machines to trust it.


Setting up your own binary cache is straightforward: https://nixos.org/nix/manual/#sec-sharing-packages

Is using Artifactory or Nexus a requirement? Why?


Mainly concerns like access control, and auditing for internal projects. If we're going to package for Nix, we need to host those packages internally somewhere, and access to that host needs to be audited/monitored. Access should be limited to developers and forbidden to non-engineering staff, etc.


Nix package definitions sit in a Git (or other VCS) repo, which you can access-control through your usual means.

The binary artifacts are behind an HTTP server, which you can access control through whatever normal access control mechanisms, or leave uncontrolled and rely on the fact that it's not possible to retrieve artifacts from the server without having access to all the build inputs so you can hash them.


Other than integration with the rest of the Nix/NixOS ecosystem, how is NixOps different from Puppet?

Reading briefly through the examples, it seems very oriented towards provisioning machine instances rather than provisioning behavior across networks or multi-machine services, which is something that Puppet really excels at. What features does this offer in addition to ones that Puppet provides?

That said, I've been getting tired of Puppet recently for some unrelated reasons, so I'll definitely give this a try.


Linux wasn't designed to be an immutably-configured operating system. When systems like Puppet try to apply configuration to a target server, they're operating on top of a layer which is inherently stateful, which has a propensity to cause all sorts of problems. What if somebody manually entered this server and screwed with the configuration, outside of Puppet? What if poorly packaged applications don't cleanly uninstall or upgrade properly? Systems like Puppet tend to just throw an error at the sysadmin and walk away.

NixOS treats immutable configuration as a first-class citizen at the OS level. Dependenices are not installed into /usr or /etc, rather specific versions of packages (identified by their hashes) are installed into /nix/store and then symlinked back to /usr or /etc. Different applications that require different versions of the same dependency, when packaged according to standard, harmoniously co-exist on the same server because the Nix packaging system symlinks the dependencies between each other in /nix/store.

You don't realize the full power of this setup until you make a mistake, make your system unbootable, and calmly tell GRUB to just use the last valid configuration - a few minutes later and you're back at a desktop. Conventional configuration management tool like Puppet tend to be worthless in those kinds of failure modes, and most sysadmins would be implementing backup restoration procedures.


Here's another example that is simple, but one I still appreciate: I wanted a slightly newer kernel. How do I get it, and ensure all the "downstream" dependencies are rebuilt (e.g. kernel modules)?[1]

I write this into my configuration.nix:

    boot.kernelPackages = pkgs.linuxPackages_latest;
I run `nixos-rebuild switch && reboot` and I'm done. I can switch back to the last one if I wanted, but this new one takes place immediately. I just did this on a server 20 minutes ago.

There are other small things I can do with this. What if I didn't want to boot this kernel to boot, just test it?

    $ nixos-rebuild build-vm
    $ ./result/bin/run-*-vm
This will instantly drop you into a QEMU-KVM instance that is using NixOS with your `configuration.nix`, but if you rebooted nothing would change. (You can also specify a different configuration.nix to test other machines![2]) Or I could boot it, but just once and switch back pretty easily if something was wrong.

---

Another: What if I wanted Tarsnap backups of something? (This may be out of date, but it's my old config, and I wrote the Tarsnap module)

      services.tarsnap.enable = true;
      services.tarsnap.archives =
        { nixos =
          { checkpointBytes = "5GB";
            directories =
            [ "/root" "/etc/nixos"
            ];
          };

          home =
          { excludes = [ "*.o" "*.hi" ];
            checkpointBytes = "5GB";
            directories =
            [ "/home/a"
            ];
          };
        };
Now I can do things like this:

    $ systemctl start tarsnap@home
    $ systemctl start tarsnap@nixos
To start on demand. Or I can configure timers/cron to do this manually.

Lots of other small things. Binary caches, your own Hydra, remote/multi-architecture builds, etc. One time I "configured" my NixOS laptop install before getting my laptop: on a remote testing server, over remote KVM. Pushed it to git. I cloned my configuration.nix during NixOS installation on my laptop a few days later, and I replicated my laptop in 10 minutes. :) Combine this with Hydra (CI) server: You can even write tests and have your laptop configuration continuously integrated all the time?!?!?!?

That said NixOS isn't always the easiest thing to use, but it does come with some big advantages!

[1] I'm aware of DKMS (and have even worked on modules using it where it was fine) but I've still had it break once or twice. Maybe that was VMWare's fault :)

[2] If you share code between all your configuration.nix files, this becomes a lot easier.


> Other than integration with the rest of the Nix/NixOS ecosystem

Nix/NixOS is the big deal actually. NixOps is just icing on the cake. A NixOS installation is configured by building/copying over a system profile, and then activating it.

With just NixOS, you edit a configuration file on the server, then run a command that will build a system profile from that configuration and activate it.

With NixOps, you have configuration files on your workstation, and run commands to build the system profiles, copy it to the servers, and activate them. Of course you can do all of this without NixOps. It just makes certain things easier.

Puppet and others can do all of this, too. However state transition tend to be the weakest link (e.g. tell Puppet to provision a brand new server vs one that has run some other stuff, how sure are you that the resulting systems are the same?). Nix/NixOS takes care of that (most of the time anyway -- there are still things like data migrations that applications have to take care of).


To say what my sibling comments say, more briefly:

With the nix system you can fully declaratively state your entire fleet of machines, from your highest level application configs all the way down to kernel patches.

When you run nixops, your machines become exactly as specified.

That this works reliably is a novelty: most other configuration management tools work like "action runners" that try in a best-effort fashion to achieve the desired state, but that doesn't always work (for example in Ansible, declaring a package as installed installs it, but NOT declaring a package will not uninstall it).

It is made possible because the nix system is designed from the bottom up to be declarative. That makes it different from Puppet.


> most other configuration management tools work like "action runners" that try in a best-effort fashion to achieve the desired state, but that doesn't always work (for example in Ansible, declaring a package as installed installs it, but NOT declaring a package will not uninstall it).

Most other configuration management tools have no choice. I got my own [1] to the point where it cannot only provision files/users/groups, but also clean them up correctly most of the time. However, since it is not the final authority (the package manager is... or rather, root is), there will always be situations where the entity that needs to be provisioned will be found to be in an entirely unanticipated state.

You have basically two ways to solve this conundrum:

1. just throw everything away and start from a clean slate when the configuration changes (aka containers)

2. do configuration management in the package manager and encourage/force root to keep his filthy hands off the system partition (aka NixOS et al)

[1] https://github.com/holocm/holo


BOSH uses both tactics. It deals in whole VMs, but parks deployed software in a known location that's the same for all stemcells.


NixOS is the system configuration layer and closer to puppet in solo mode. With a single (or multiple) nix file(s) it's possible to configure a machine and it's services entirely.

NixOps is the cloud configuration layer, it provisions cloud resources and uploads NixOS configuration to the machines. It's closer to terraform in that regard. It also has a state file (currently sqlite3) and uses a push model to propagate configuration changes.

What's missing right now is a third option where an agent is installed on the machines for a more dynamic configuration of the cluster as a whole. This role will probably be filled by Kubernetes or Nomad.


Note that with nixops, you can pass variables to the whole deployment configuration, and as such, your deployment is "dynamic" if you reconfigure it often enough :)

Here's a deployment of mesos where you can specify how many slaves and masters you want: https://github.com/wmertens/nixops-mesos


Is there good explanation anywhere of the syntax of those files? I tried getting into nix multiple times and was always put off by the lack of understandability of this.


Nix is a bit like JSON + functions.

An object (called attrset in nix):

    { a = 3; b = 4; }
A list:

    [ 1 2 3 4 ];
A string:

     "hello"
A multi-line string:

    ''
      multine
      string
    ''
This defines a lambda with two arguments:

    a: b: a + b;
It's possible to pattern-match arguments. This accepts an attrset with keys a and b, and maps them to local variables:

    { a, b }: a + b;
There are some variations around that like `{ a ? 3 }:` `{ a }@attrs:` and `{ a, ... }`

One special type is the path, which has it's own literal that starts with `./`

To load a file, the `import` keyword is being used. In most cases the whole file is a big lambda function, which gets returned by the import.

Once you know that you can start reading nixpkgs and learn about the building blocks to constructs packages.


Also, for convenience, nested objects/attrsets can be shortened. For instance:

    {
        a = {
            b = {
                c = 3;
            };
        };
    }
can be shortened to:

    {
        a.b.c = 3;
    }
This is used quite often in NixOS system configuration files. There you'll find lines like:

    services.openssh.enable = true;
For a list of all system configuration options that NixOS supports by default see (NixOS options)[https://nixos.org/nixos/options.html].


I read Nix Pills and they were very informative -- https://nixos.org/nixos/nix-pills/



The syntax is, by far, the most painful part. Change the drop-down to Nix then click on the manual. The website is hard to navigate in that way.


https://learnxinyminutes.com/docs/nix/

Also includes links to the only other resources I found.




NixOS makes a great OS for personal/workstation use, too.

With NixOS, my system is always 100% clean, and never broken.

I can install packages in my home directory without root.

Best of all, whenever I want to write any code, I can use nix-shell to make the toolchain and dependencies temporarily available to a self-contained shell. No more cluttering my system with libfoobar-dev for a quick build, and forgetting why.


I would love if someone could compare NixOps with BOSH.


I haven't done much Nix, but I do peek over the fence now and then because I (and others) wonder about applying it to buildpacks and/or OCI layer construction. I do a lot more BOSH, since I work at Pivotal. So these remarks are probably wrong.

However, from my ignorant viewpoint:

The deep agreement is that machine-level statefulness is evil. Trying to repair it in-situ is pointless when you can simply reset to known-good states.

BOSH does this at the whole-machine level. You get a stemcell (fixed target), you get releases (fixed configuration of software) and you get deployments (fixed mappings from releases into running systems).

When you make a change, you redeploy. BOSH recompiles any changed release packages and then rolls them out across the fleet, starting with canaries.

BOSH lets you define the number of VMs, networks and whatnot; I'm not sure if NixOps does.

Treating components as replaceable is the main point of departure from previous generations of configuration management, which largely evolved out of a world in which the mission was "make this single machine look right, starting from an unknown state".

NixOps appears to do something similar. I'm unclear as to whether it operates at the whole-machine level (à la BOSH) or whether it can perform transformations within an existing machine. The latter would, obviously, be a lot faster than replacing VMs wholesale.

That said, whole-machine replacement does have one accidental benefit: it's ruthless. You very quickly encounter the one-in-a-million edge cases for running services because BOSH will quite unmercifully steamroll VMs during a deploy. Any stateful distributed services that aren't ready to have running machines yanked from under them at short notice will quickly reveal themselves.

You'd be surprised how many systems that advertise themselves as tolerant to random machine loss ... er ... aren't. Not sure much how I can go into this, but some distributed systems do much more poorly than the stuff on the tin suggests, even after substantial engineering on our part to try and prop them up.

Disclosure: I work for Pivotal, so I use BOSH sometimes in my dayjob. But I'm by no means a capable operator or BOSH expert.


* NixOps lets you define the complete layout of your deployment (in Nix, obviously): how many VMs, what they should be running, where they should be running.

* NixOps works on whole machine + transformation of machine level: NixOps builds all system configurations of the whole network, then makes sure the required target hosts are available, then copies the builds to the hosts, and then atomically switches all hosts to the new configuration at once. NixOS is stateless, and a new config only means that the new service configurations are applied.

Rollback is a matter of switching all VMs back to their previous configurations, takes less than a minute in all.


Thankyou for the additional detail. I'd say that the two arrived at similar outcomes, which isn't surprising given NixOps's roots.


NixOS/NixOps will manage the whole machine, but it will replace individual services instead of rebuilding the whole VM.


That would be interesting indeed!


Seems neat, better than Terraform, although limited to just nix and a few providers. Hopefully that will grow. I've been working on a provisioning tool myself. Provisioning is hard, and supporting multiple APIs can be really difficult.


I don't really see Nix as a limitation. It has a good claim to being a universal system configuration tool. Or how do you mean it's limited?


When I last tried it about 6 months ago it seemed to be a complete mess. I failed to get any recent ruby gems installed. There were dozens of different articles describing how to work with package repositories, and all of them seemed to be outdated by varying degrees. I also got the feeling that nix had more of a focus on desktop systems, for whatever reason.


I use it as a system level configuration tool, and then use Docker for the application level. This is not how the NixOS sees itself ideally, but in my experience their ideology does not map well to the reality of application deployment.

So what I do is have all system level tools installed using NixOS: the OS, logging, systemd, Docker, sshd and so on. Then I have my applications built as Docker containers and provision them as systemd units also configured by NixOS files.


That sounds reasonable, but it kinda makes me question NixOS' value proposition. It seems that any one of the current distributions works well as a generic base for containers.

There are probably lots of systems running standard Linux packages with more complex configurations (i.e. Mailservers, maybe databases, proxies etc) where I could imagine it being useful? Or maybe their desktop work makes more sense than I thought–my desktop environment is much more customised than any server, and I've never tried if the setup scripts I try to maintain actually work.


I don't think any operating system at this moment does well as a base for containers. At least not from the box, and as soon as you have to customise the OS you need tooling like Puppet/Chef/Salt to make your customizations reproducible. And that's the exact value proposition of NixOS, it's not only reproducible, the OS will be a complete reflection of your Nix specifications from the ground up.

I feel NixOS definitely gets some stuff wrong, and I hope at some point it gets some good competition. Mainly because the Nix language looks a bit awkward to me, and also because I don't fully get their package management situation. But the idea of having the operating system be configured from a specification is just a level up from old fashioned linux distros.


I thought the same as well, but recently I've been having a decent go at it with RancherOS. Everything boots from two docker processes, either at the system or user level. Almost anything in the system can be pretty easily configured via cloud-init.yml file. If you want an Ubuntu console layer(or Alpine) just specify it in the yml.

It's great to see NixOS and similar tools gaining ground. But personally I find that docker's configurations simpler to grok than NixOS's syntax. Plus it's interesting the granular level of control over system security and system access gaiened via docker (really mostly Linux secomp controls and process namespaces). It's kind of like having capabilities built into the OS on top of specification driven system configuration.


Just some notes you might be interested in:

NixOS is capable of creating a Docker image given a Nix specification, by turning the Nix store into an archive. These images are quite minimal since they contain exactly the dependencies needed for your app.

NixOS also has its own container system, so your server configuration can simply specify a subconfiguration that will be spawned as a container. The containers reuse the Nix store from the host system, which means you get a fine-grained caching system.

I haven't used those things much so I don't really know how production-ready they are.

I sometimes use Docker (with Dockerfiles or DockerHub images) on NixOS as a way to run systems that I don't want to package with Nix. I've made a little Nix module for declaring Docker services as systemd services.

That's part of why I don't feel like Nix is a limitation. I have actually used NixOps to deploy NixOS configurations that include Docker services, so my feeling of Nix is that it kind of supervenes on everything else...


Well, Nix is probably never going to be usable by non-developers and is very far from production quality for any kind of infrastructure. I mean theoretically you can use it in production if you avoid nixpkgs and make your own package repository with your own packages and your own tools. Playing with it on desktop is fine, although you can't get far without learning the language, reading sources and doing some functional programming.

Still, Nix should be at least an inspiration for next real world package managers.


Can I use it to deploy either client or server applications on windows? All configuration and deployment tools I've looked at seem like they assume the only deployment target is linux servers running web applications. I don't care about the web. I need a deployment tool that works for local research on my clients' analysts' desktops, not just devops on remote somewhere.


BOSH works with Windows machines.

Most of the pain is building stemcells (the target disk image). Microsoft haven't quite made their peace with the idea.


Have you looked into the Windows features of Ansible or SaltStack? Would be interested to hear about experiences with them.


I would like to adopt something for configuration of my personal machines. Desktop and tablet running Windows 10, laptop running macOS.

If possible I'd prefer to use one tool for both OSes.


Nix supposedly works in the new Linux subsystem of Windows 10, but I haven't tried it.


Nix itself does indeed work on the latest version of WSL. However, there are still issues with building most of the packages from nixpkgs. For instance, I'm still waiting for this issue to be solved: https://github.com/Microsoft/BashOnWindows/issues/1878

Other distros are also affected by these issues, so I think they'll be solved eventually.


The `none` provider works for any servers that has an ssh connection. I use it myself to maintain my server on vultr.com . It has been pretty neat!


So put on a Nix overlay on your Linux/Unix flavor of choice like people have done with pkgsrc in and outside the NetBSD community?


I don't know how NetBSD handles this, but for any provider that doesn't allow you to use custom ISO's or ipxe, you can use [nixos-in-place](https://github.com/jeaye/nixos-in-place). I've used it before. It isn't ideal, but it gets the job done.


No. Vultr supports custom ISO's. And I uploaded the nixos image to my account


FYI, there's a WIP PR for Vultr backend support in NixOps.

https://github.com/NixOS/nixops/pull/634

I cherry-picked this into my local NixOps repo and have been using it for several weeks to deploy to Vultr servers with no issues, except one -- the author has hard-wired an assumption that you're using Btrfs for your root filesystem. I didn't want that so I simply replaced that hard-wired assumption with ext4 in my personal branch. Now everything is working great.


I'm the author of that PR. I apologize for the forced usage of btrfs. This was one of my first delves into the nix language, and I literally copied the digital ocean backend and changed the API calls (and the d.o had a hardcoded filesystem and I wanted btrfs and didn't know how at the time to move that into the configuration level). After the 17.09 release of NixOS is completed, I'll put some focus back in that PR and hopefully get it merged upstream.


No need to apologize. It works great for my needs and it was easy enough just to hack your hack :)


Can anyone comment on whether or not Nix is 100% reproducible? To the best of my limited understanding, it is not.


Nix doesn't guarantee bit-for-bit reproducibility: That is, two different builds of a Nix package may not produce bit-identical output. (They probably will, because of all the sandboxing, but nothing prevents an arbitrary package from generating random numbers at build time.)

Certainly Nix makes it a lot easier to get reproducibility in practice, though.


It is not. In this talk [1] Eelco Dolstra touches upon that a bit also.

[1] https://www.youtube.com/watch?v=QvH5qU1qBXY&feature=youtu.be




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: