In view of some replies downthread: yes, application-level management is entirely possible with Nix(OS).
Currently, this works best for Haskell, for which there is a sizeable amount (relatively speaking) of Nix tooling available. This is largely because a lot of Nix(OS) users also use Haskell, perhaps because lend themselves to very similar modes of "declarative" thinking.
This is a thorough, excellent guide on how to develop Haskell projects with Nix, including managing non-Haskell dependencies (zlib, SDL, ncurses, and so on):
I only started using NixOS about a month ago, and it's fantastic. The best part is that binary caches are available for almost everything, so the hitherto-painful step where one compiles Haskell dependencies locally has been completely eliminated. The best way I can describe how I work now is "Python virtualenvs, but for everything at once and just so much better".
PS. Gabriel Gonzalez is, in general, someone who makes great documenation. Check out the tutorial for his stream-processing library Pipes: the ASCII art alone is awesome :)
We use NixOS in production in our product, for our cloud services, and for some development environments.
We build everything with nix from sbt projects, go projects, haskell projects, purescript projects, docker images, and npm projects.
It's an incredible tool.
NixOps is also a good tool but there are a lot of issues with it we haven't had time to contribute fixes for. We also wanted it to be lighter weight and to decouple system specification from system deployment. So I wrote a tool that handles performing the same system deployment operations that NixOps does for you and you can simply pipe the nix path result from nix-build to it. We use terraform to specify and manage the cloud resources.
We will definitely be publishing it. Time is very short and the backlog of projects to publish is growing though so it might take time to polish and write a blogpost and tutorial.
I work for Awake Security so keep an eye out on the company's GitHub org as we've been publicly releasing a lot of projects even though they aren't completely polished or announced yet. I'll see if I can do this for our deploy tool soon.
Basically the kind of person I would love to buy a beer any time because he does so much great work.
on topic: The binary caches are great.
For work I'm currently evaluating https://github.com/typeable/stackage2nix and https://github.com/input-output-hk/stack2nix to generate nix expressions from our haskell stack projects. And then push the build artifacts to a shared cache. Such that nobody has to ever compile a dependency twice. Because nothing sucks more than starting your day watching the compiler churn away at compiling after version bumps, to then also see a collegue go to the same pain all over again.
I love Nix concepts and all it's subprojects (NixOS, NixOps, Hydra, Disnix).
However, i've found NixPkgs to be a bit messy. It's too big, targets too many platforms and it's too disorganised. But to be completely fair, I think it's getting a lot of good refactorings lately.
For large enterprise use, Nix is going to need to mature to a point where it's possible for enterprises to host a Nixpkgs cache locally, using a binary repository manager like Artifactory or Nexus, both for sheer cost (to save on bandwidth of thousands of servers downloading updates from the Internet) as well as risk reduction (in case the public binary caches are unavailable at critical moments). If this is doable today, it's not really documented.
I'm sure the project will get there... eventually.
You can install nix-serve on any machine to share its binary cache (including patched applications). Then add all of the enabled machines to every machine's nix.binaryCaches in order of preference. If they don't have what you need, it will fall back to the upstream server. You'll want to make sure they are upgraded in an appropriate order to minimize the need to leave your network for updates. It's true this isn't well documented, but I figured it out in about a day and was surprised at how simple it turned out to be.
You can even go further and set up your own build server to service all the architectures you need to support.
I found Guix a lot more organised. Most package code is very simple, and the toolchain is also quite elegant. Hopefully the latter will be addressed by the new nix interface.
Mainly concerns like access control, and auditing for internal projects. If we're going to package for Nix, we need to host those packages internally somewhere, and access to that host needs to be audited/monitored. Access should be limited to developers and forbidden to non-engineering staff, etc.
Nix package definitions sit in a Git (or other VCS) repo, which you can access-control through your usual means.
The binary artifacts are behind an HTTP server, which you can access control through whatever normal access control mechanisms, or leave uncontrolled and rely on the fact that it's not possible to retrieve artifacts from the server without having access to all the build inputs so you can hash them.
Other than integration with the rest of the Nix/NixOS ecosystem, how is NixOps different from Puppet?
Reading briefly through the examples, it seems very oriented towards provisioning machine instances rather than provisioning behavior across networks or multi-machine services, which is something that Puppet really excels at. What features does this offer in addition to ones that Puppet provides?
That said, I've been getting tired of Puppet recently for some unrelated reasons, so I'll definitely give this a try.
Linux wasn't designed to be an immutably-configured operating system. When systems like Puppet try to apply configuration to a target server, they're operating on top of a layer which is inherently stateful, which has a propensity to cause all sorts of problems. What if somebody manually entered this server and screwed with the configuration, outside of Puppet? What if poorly packaged applications don't cleanly uninstall or upgrade properly? Systems like Puppet tend to just throw an error at the sysadmin and walk away.
NixOS treats immutable configuration as a first-class citizen at the OS level. Dependenices are not installed into /usr or /etc, rather specific versions of packages (identified by their hashes) are installed into /nix/store and then symlinked back to /usr or /etc. Different applications that require different versions of the same dependency, when packaged according to standard, harmoniously co-exist on the same server because the Nix packaging system symlinks the dependencies between each other in /nix/store.
You don't realize the full power of this setup until you make a mistake, make your system unbootable, and calmly tell GRUB to just use the last valid configuration - a few minutes later and you're back at a desktop. Conventional configuration management tool like Puppet tend to be worthless in those kinds of failure modes, and most sysadmins would be implementing backup restoration procedures.
Here's another example that is simple, but one I still appreciate: I wanted a slightly newer kernel. How do I get it, and ensure all the "downstream" dependencies are rebuilt (e.g. kernel modules)?[1]
I write this into my configuration.nix:
boot.kernelPackages = pkgs.linuxPackages_latest;
I run `nixos-rebuild switch && reboot` and I'm done. I can switch back to the last one if I wanted, but this new one takes place immediately. I just did this on a server 20 minutes ago.
There are other small things I can do with this. What if I didn't want to boot this kernel to boot, just test it?
$ nixos-rebuild build-vm
$ ./result/bin/run-*-vm
This will instantly drop you into a QEMU-KVM instance that is using NixOS with your `configuration.nix`, but if you rebooted nothing would change. (You can also specify a different configuration.nix to test other machines![2]) Or I could boot it, but just once and switch back pretty easily if something was wrong.
---
Another: What if I wanted Tarsnap backups of something? (This may be out of date, but it's my old config, and I wrote the Tarsnap module)
To start on demand. Or I can configure timers/cron to do this manually.
Lots of other small things. Binary caches, your own Hydra, remote/multi-architecture builds, etc. One time I "configured" my NixOS laptop install before getting my laptop: on a remote testing server, over remote KVM. Pushed it to git. I cloned my configuration.nix during NixOS installation on my laptop a few days later, and I replicated my laptop in 10 minutes. :) Combine this with Hydra (CI) server: You can even write tests and have your laptop configuration continuously integrated all the time?!?!?!?
That said NixOS isn't always the easiest thing to use, but it does come with some big advantages!
[1] I'm aware of DKMS (and have even worked on modules using it where it was fine) but I've still had it break once or twice. Maybe that was VMWare's fault :)
[2] If you share code between all your configuration.nix files, this becomes a lot easier.
> Other than integration with the rest of the Nix/NixOS ecosystem
Nix/NixOS is the big deal actually. NixOps is just icing on the cake. A NixOS installation is configured by building/copying over a system profile, and then activating it.
With just NixOS, you edit a configuration file on the server, then run a command that will build a system profile from that configuration and activate it.
With NixOps, you have configuration files on your workstation, and run commands to build the system profiles, copy it to the servers, and activate them. Of course you can do all of this without NixOps. It just makes certain things easier.
Puppet and others can do all of this, too. However state transition tend to be the weakest link (e.g. tell Puppet to provision a brand new server vs one that has run some other stuff, how sure are you that the resulting systems are the same?). Nix/NixOS takes care of that (most of the time anyway -- there are still things like data migrations that applications have to take care of).
To say what my sibling comments say, more briefly:
With the nix system you can fully declaratively state your entire fleet of machines, from your highest level application configs all the way down to kernel patches.
When you run nixops, your machines become exactly as specified.
That this works reliably is a novelty: most other configuration management tools work like "action runners" that try in a best-effort fashion to achieve the desired state, but that doesn't always work (for example in Ansible, declaring a package as installed installs it, but NOT declaring a package will not uninstall it).
It is made possible because the nix system is designed from the bottom up to be declarative. That makes it different from Puppet.
> most other configuration management tools work like "action runners" that try in a best-effort fashion to achieve the desired state, but that doesn't always work (for example in Ansible, declaring a package as installed installs it, but NOT declaring a package will not uninstall it).
Most other configuration management tools have no choice. I got my own [1] to the point where it cannot only provision files/users/groups, but also clean them up correctly most of the time. However, since it is not the final authority (the package manager is... or rather, root is), there will always be situations where the entity that needs to be provisioned will be found to be in an entirely unanticipated state.
You have basically two ways to solve this conundrum:
1. just throw everything away and start from a clean slate when the configuration changes (aka containers)
2. do configuration management in the package manager and encourage/force root to keep his filthy hands off the system partition (aka NixOS et al)
NixOS is the system configuration layer and closer to puppet in solo mode. With a single (or multiple) nix file(s) it's possible to configure a machine and it's services entirely.
NixOps is the cloud configuration layer, it provisions cloud resources and uploads NixOS configuration to the machines. It's closer to terraform in that regard. It also has a state file (currently sqlite3) and uses a push model to propagate configuration changes.
What's missing right now is a third option where an agent is installed on the machines for a more dynamic configuration of the cluster as a whole. This role will probably be filled by Kubernetes or Nomad.
Note that with nixops, you can pass variables to the whole deployment configuration, and as such, your deployment is "dynamic" if you reconfigure it often enough :)
Is there good explanation anywhere of the syntax of those files? I tried getting into nix multiple times and was always put off by the lack of understandability of this.
NixOS makes a great OS for personal/workstation use, too.
With NixOS, my system is always 100% clean, and never broken.
I can install packages in my home directory without root.
Best of all, whenever I want to write any code, I can use nix-shell to make the toolchain and dependencies temporarily available to a self-contained shell. No more cluttering my system with libfoobar-dev for a quick build, and forgetting why.
I haven't done much Nix, but I do peek over the fence now and then because I (and others) wonder about applying it to buildpacks and/or OCI layer construction. I do a lot more BOSH, since I work at Pivotal. So these remarks are probably wrong.
However, from my ignorant viewpoint:
The deep agreement is that machine-level statefulness is evil. Trying to repair it in-situ is pointless when you can simply reset to known-good states.
BOSH does this at the whole-machine level. You get a stemcell (fixed target), you get releases (fixed configuration of software) and you get deployments (fixed mappings from releases into running systems).
When you make a change, you redeploy. BOSH recompiles any changed release packages and then rolls them out across the fleet, starting with canaries.
BOSH lets you define the number of VMs, networks and whatnot; I'm not sure if NixOps does.
Treating components as replaceable is the main point of departure from previous generations of configuration management, which largely evolved out of a world in which the mission was "make this single machine look right, starting from an unknown state".
NixOps appears to do something similar. I'm unclear as to whether it operates at the whole-machine level (à la BOSH) or whether it can perform transformations within an existing machine. The latter would, obviously, be a lot faster than replacing VMs wholesale.
That said, whole-machine replacement does have one accidental benefit: it's ruthless. You very quickly encounter the one-in-a-million edge cases for running services because BOSH will quite unmercifully steamroll VMs during a deploy. Any stateful distributed services that aren't ready to have running machines yanked from under them at short notice will quickly reveal themselves.
You'd be surprised how many systems that advertise themselves as tolerant to random machine loss ... er ... aren't. Not sure much how I can go into this, but some distributed systems do much more poorly than the stuff on the tin suggests, even after substantial engineering on our part to try and prop them up.
Disclosure: I work for Pivotal, so I use BOSH sometimes in my dayjob. But I'm by no means a capable operator or BOSH expert.
* NixOps lets you define the complete layout of your deployment (in Nix, obviously): how many VMs, what they should be running, where they should be running.
* NixOps works on whole machine + transformation of machine level: NixOps builds all system configurations of the whole network, then makes sure the required target hosts are available, then copies the builds to the hosts, and then atomically switches all hosts to the new configuration at once. NixOS is stateless, and a new config only means that the new service configurations are applied.
Rollback is a matter of switching all VMs back to their previous configurations, takes less than a minute in all.
Seems neat, better than Terraform, although limited to just nix and a few providers. Hopefully that will grow. I've been working on a provisioning tool myself. Provisioning is hard, and supporting multiple APIs can be really difficult.
When I last tried it about 6 months ago it seemed to be a complete mess. I failed to get any recent ruby gems installed. There were dozens of different articles describing how to work with package repositories, and all of them seemed to be outdated by varying degrees. I also got the feeling that nix had more of a focus on desktop systems, for whatever reason.
I use it as a system level configuration tool, and then use Docker for the application level. This is not how the NixOS sees itself ideally, but in my experience their ideology does not map well to the reality of application deployment.
So what I do is have all system level tools installed using NixOS: the OS, logging, systemd, Docker, sshd and so on. Then I have my applications built as Docker containers and provision them as systemd units also configured by NixOS files.
That sounds reasonable, but it kinda makes me question NixOS' value proposition. It seems that any one of the current distributions works well as a generic base for containers.
There are probably lots of systems running standard Linux packages with more complex configurations (i.e. Mailservers, maybe databases, proxies etc) where I could imagine it being useful? Or maybe their desktop work makes more sense than I thought–my desktop environment is much more customised than any server, and I've never tried if the setup scripts I try to maintain actually work.
I don't think any operating system at this moment does well as a base for containers. At least not from the box, and as soon as you have to customise the OS you need tooling like Puppet/Chef/Salt to make your customizations reproducible. And that's the exact value proposition of NixOS, it's not only reproducible, the OS will be a complete reflection of your Nix specifications from the ground up.
I feel NixOS definitely gets some stuff wrong, and I hope at some point it gets some good competition. Mainly because the Nix language looks a bit awkward to me, and also because I don't fully get their package management situation. But the idea of having the operating system be configured from a specification is just a level up from old fashioned linux distros.
I thought the same as well, but recently I've been having a decent go at it with RancherOS. Everything boots from two docker processes, either at the system or user level. Almost anything in the system can be pretty easily configured via cloud-init.yml file. If you want an Ubuntu console layer(or Alpine) just specify it in the yml.
It's great to see NixOS and similar tools gaining ground. But personally I find that docker's configurations simpler to grok than NixOS's syntax. Plus it's interesting the granular level of control over system security and system access gaiened via docker (really mostly Linux secomp controls and process namespaces). It's kind of like having capabilities built into the OS on top of specification driven system configuration.
NixOS is capable of creating a Docker image given a Nix specification, by turning the Nix store into an archive. These images are quite minimal since they contain exactly the dependencies needed for your app.
NixOS also has its own container system, so your server configuration can simply specify a subconfiguration that will be spawned as a container. The containers reuse the Nix store from the host system, which means you get a fine-grained caching system.
I haven't used those things much so I don't really know how production-ready they are.
I sometimes use Docker (with Dockerfiles or DockerHub images) on NixOS as a way to run systems that I don't want to package with Nix. I've made a little Nix module for declaring Docker services as systemd services.
That's part of why I don't feel like Nix is a limitation. I have actually used NixOps to deploy NixOS configurations that include Docker services, so my feeling of Nix is that it kind of supervenes on everything else...
Well, Nix is probably never going to be usable by non-developers and is very far from production quality for any kind of infrastructure. I mean theoretically you can use it in production if you avoid nixpkgs and make your own package repository with your own packages and your own tools. Playing with it on desktop is fine, although you can't get far without learning the language, reading sources and doing some functional programming.
Still, Nix should be at least an inspiration for next real world package managers.
Can I use it to deploy either client or server applications on windows? All configuration and deployment tools I've looked at seem like they assume the only deployment target is linux servers running web applications. I don't care about the web. I need a deployment tool that works for local research on my clients' analysts' desktops, not just devops on remote somewhere.
Nix itself does indeed work on the latest version of WSL. However, there are still issues with building most of the packages from nixpkgs. For instance, I'm still waiting for this issue to be solved: https://github.com/Microsoft/BashOnWindows/issues/1878
Other distros are also affected by these issues, so I think they'll be solved eventually.
I don't know how NetBSD handles this, but for any provider that doesn't allow you to use custom ISO's or ipxe, you can use [nixos-in-place](https://github.com/jeaye/nixos-in-place). I've used it before. It isn't ideal, but it gets the job done.
I cherry-picked this into my local NixOps repo and have been using it for several weeks to deploy to Vultr servers with no issues, except one -- the author has hard-wired an assumption that you're using Btrfs for your root filesystem. I didn't want that so I simply replaced that hard-wired assumption with ext4 in my personal branch. Now everything is working great.
I'm the author of that PR. I apologize for the forced usage of btrfs. This was one of my first delves into the nix language, and I literally copied the digital ocean backend and changed the API calls (and the d.o had a hardcoded filesystem and I wanted btrfs and didn't know how at the time to move that into the configuration level). After the 17.09 release of NixOS is completed, I'll put some focus back in that PR and hopefully get it merged upstream.
Nix doesn't guarantee bit-for-bit reproducibility: That is, two different builds of a Nix package may not produce bit-identical output. (They probably will, because of all the sandboxing, but nothing prevents an arbitrary package from generating random numbers at build time.)
Certainly Nix makes it a lot easier to get reproducibility in practice, though.
Currently, this works best for Haskell, for which there is a sizeable amount (relatively speaking) of Nix tooling available. This is largely because a lot of Nix(OS) users also use Haskell, perhaps because lend themselves to very similar modes of "declarative" thinking.
This is a thorough, excellent guide on how to develop Haskell projects with Nix, including managing non-Haskell dependencies (zlib, SDL, ncurses, and so on):
https://github.com/Gabriel439/haskell-nix/
I only started using NixOS about a month ago, and it's fantastic. The best part is that binary caches are available for almost everything, so the hitherto-painful step where one compiles Haskell dependencies locally has been completely eliminated. The best way I can describe how I work now is "Python virtualenvs, but for everything at once and just so much better".
PS. Gabriel Gonzalez is, in general, someone who makes great documenation. Check out the tutorial for his stream-processing library Pipes: the ASCII art alone is awesome :)
https://hackage.haskell.org/package/pipes-4.3.4/docs/Pipes-T...