Can someone explain to me the benefits of moving to chef or puppet as opposed to say my current approach of writing deploy scripts in pythons fabric (basically ssh and file operations library)
I have even used Puppet but I could not get over the feeling that it is more complex than I would imagine for small end users (I have managed upto 40 servers with my libraries - it seemed quite feasible to double that. Maybe it's me but 100 servers sounds a lot and is manageable (deployment wise) with controlled scripts.
Configuration management is one of those things where you reap most of the benefit just by doing it, not by doing it with a particular tool.
That said, I prefer Puppet because its abstractions make more sense to me, and are pretty easily extended using Ruby.
Puppet is also trivial to bootstrap (at least on Debian). Unlike other Ruby stacks, I can install it using the package manager and be up and running in no time. With Chef you have to maintain additional services like CouchDB and RabbitMQ.
The real reason not to use ad hoc scripts or a tool you wrote yourself is that things like cfengine, Chef, and Puppet are maintained and used by communities -- your successor(s) will thank you.
> With Chef you have to maintain additional services like CouchDB and RabbitMQ.
And care about a lot of dependencies. Installing chef-server on a fresh ubuntu quantal:
0 upgraded, 444 newly installed, 0 to remove and 58 not upgraded.
Need to get 280 MB of archives.
After this operation, 775 MB of additional disk space will be used.
Of course a lot of that is due to how ubuntu packages java, but still... that's quite a lot of software to care about.
With Chef 11 Server, the CouchDB component is removed for PostgreSQL. RabbitMQ is still there. Opscode provides a single package with everything above libc required to run the server.
A large part of that was an unfortunate side-affect of gecode (a library Chef used for dependency resolution in Chef 10) not having a -nox package, thus pulling in libqt4-dev, which then spirals out of control. The chef-server cookbook used the gecode cookbook which automated installing from source without qt to avoid this if one desired.
> With Chef you have to maintain additional services like CouchDB and RabbitMQ.
You can avoid those dependencies by using Chef Solo instead of Chef Server. In my opinion chef-solo is also a much easier way to learn Chef, so you can get the main concepts down before tackling chef-server.
But yes, Chef's dependencies are staggering. I once tried using an AWS micro instance for a Chef server, and it kept crashing because the system didn't have enough resources.
It's helpful to think of two separate processes: managing the server that will host your application(s) and deploying to the servers.
SSH + shell scripts are pretty good for one off things. But they're rarely idempotent. Chef and Puppet will say "you asked for nginx version x, you have it, so I won't bother installing it". You can also handle dependencies and roles, so you end up with things like "this hosts a Rails application, so you need Ruby and bundler, nginx configured a certain way, and the upstream load balancer needs to point to the new server". So using shell scripts ends up either taking longer or as you grow it gets more complex because you have to check for different things that a specialized tool just handles natively.
On the deploy side, ssh + shell scripts might be a better solution (IMHO stuff like Capistrano and Fabric can be seen as a fancy wrapper around that). So the procedure for a deploy might be "check out source, copy to a certain folder, create these symlinks, restart these daemons". Deploys, by their very nature, aren't idempotent, as each deploy is intended to change the state, you either finish it or abort and roll back.
I interviewed community managers from Puppet and Chef over at http://linuxadminshow.com, there are some interesting comments from both about the need for it even at small scale. Nathen from Opscode had a few comments about erchef. He also runs a podcast called "Food Fight" where they very recently talked about using Chef for the deploy process (http://foodfightshow.org/2013/01/application-deployment.html)
Can someone explain to me the benefits of moving to chef or puppet as opposed to say my current approach of writing deploy scripts in pythons fabric (basically ssh and file operations library)`
I can't talk about chef but puppet basically gives you a declarative language to specify the end-state in which the server config should end up as. So you say something like "the apache server must be enabled and running" and then when the client runs it makes sure the state matches that. So if for some reason apache gets disabled (say a new version of the package disables it) the next time the client runs it will right that.
I have even used Puppet but I could not get over the feeling I was placing a lot of trust in an agglomeration of system in provided recipes
You don't have to use the recipes. I had the same worry so what I did with my own puppet usage was set a rule that I won't copy recipes from anywhere and just use puppet to do the same things I would do directly on the server but describe them using puppet so I can replicate them later. This has worked great for me. I still have to learn all the details of how to configure everything on the machine but I now have a version controlled log of how I've configured everything and I can very easily apply the same recipe between two machines.
And even though I don't use copied recipes puppet itself abstracts away a bunch of stuff. So for example "make sure apache is installed" is implemented with apt in ubuntu and rpm in redhat. So if I wanted to repurpose my recipes from ubuntu to fedora I'd have to recode paths and package names but I wouldn't have to figure out how to interact with the package manager, what the command to create a new user is, etc.
With chef/puppet/salt/... it's easier to manage things that happen to change. For example it's much easier to express "generate configuration from this template into that file and if it changes restart following services" in those systems. That's what they're designed to do and that's what requires minimal number of steps.
With fabric it's easier to say - do this, do that. Both have their own place and you can use both for different tasks.
It may not matter at the first deployment stage (it's always install this, install that), but it does make a difference when you do continuous deployment for example. That are many dependencies that you'd have to implement in details and manually with systems like fabric.
Another benefit is abstracting away the management via a boxed in webapp. I manage about 2000 servers all running chef. Things like IPs, host files, firewall settings etc, are stored in databags, each server has its own databag that can be manipulated by a Rails app/the Spice gem (thanks Dan Ryan!). As someone who is tasked with managing all of this, along with bringing the less experienced employees up to speed, it is a way for me to normalize all of these things, and abstract things away enough that almost anyone can open a port up on any server, setup a rails app etc etc. And whomever takes over after me, will just have to read one set of docs, get an overview, and they should be able to follow the design patterns of how things are setup, much more easily then random ssh/bash scripts.
Chef is an automation framework that can handle your entire infrastructure. We use it to manage thousands of servers from PXE boot, OS installation and to configure the entire software stack. It basically give you a better environment to manage your automation scripts. The cookbooks are unit of works and you have testing frameworks.
Chef and Fabric can play nice together. Use chef to configure the system but use fabric to manage them. Chef isn't fantastic at operational activities, especially those that need to be coordinated with a couple of nodes. Check out Pychef to drop your chef roles and hosts into fabric scripts.
I maintain a database of all the nodes (servers, Linux or BSD so not talking routers ) as well - one would go mad otherwise. And I feel that at certain scales one should tell hosts what to do, not rely on some auto discovery
Process - especially if that process is not out and out cloning.
Edit: just to clarify I am not attacking your choice here or your contribution to the state if the art in GitHub - we need more people making. But I would like to understand why a guy who is unlikely to run more than a couple dozen servers again needs to learn to debug and battle something like chef when the imperative scripting approach is simpler and more direct
I suppose I am asking, at what scale does chef et al pay off compared to expect-like scripts (assuming the development time of a simple script is roughly equal in both)
Then, yes, Chef is probably overkill. Although, the resources (template, command, etc...) are nice and helpful. But I wouldn't replace an existing working process with Chef is there's no obvious benefit.
Do it early, for many reasons. I worked as a systems administrator for startups before we wrote Chef, and I wouldn't go back to managing infrastructure by hand.
A few reasons:
* You don't have to maintain that database manually, Chef collects and stores node information for you.
* If you've already automated building out your servers, you're all set when you need more of them.
* The recipes document your systems, so if you "get hit by a bus" someone else can figure out how the bits get around.
* When your datacenter burns down, you can rebuild quickly from bare metal, a chef repository, and backups of your application data.
Also see this article written by Jesse Robbins, one of the Opscode founders, but before Chef existed:
> needs to learn to debug and battle something like chef when the imperative scripting approach is simpler and more direct
The biggest difference is, as the previous replies hinted at, Chef/Puppet/CFengine will each run "constantly" checking the state of the system against their rules.
With your fabric-approach you can only cause changes to happen when you're in front of the computer are you run "fab apply", "fab apt-get-upgrade", etc.
FWIW I also use a simpler alternative, recently documented, introduced, and explored here:
With the obvious caveat that you must have written the rules/policies/recipes for that to happen.
(I've seen some cfengine repositories, for example, that assume Apache is present and just start tweaking the configuration files. If you're doing it properly you'd want to define a rule initially "install apache". That's why it is often instructive to test out your rules on a pristine/virgin installation of your preferred distribution to see that it works from start to finish.)
And I feel that at certain scales one should tell hosts what to do, not rely on some auto discovery Process - especially if that process is not out and out cloning.
That's one thing, in Chef, that scares me. Access control on node attributes is inexistent. Even in private/hosted chef. So that a node can modify its own run_list, its own tags, etc...
A frontend node that gets rooted can suddenly become a backend database. If the environment is configured to allow for automated configuration of everything, this can get ugly very quickly.
Even with Chef, human approval of new nodes and roles is still mandatory.
Congratulations to the Chef crew! Here's one thing I still am wondering though, being a relatively heavy Puppet user now. What are the big benefits to using Chef, or would it be mostly just admin choice here? When I was originally deciding upon a CM system I looked at Puppet, Chef and Salt and decided that Salt was too new (even though I know Python), Chef declared everything in Ruby and I had no intention of learning Ruby just to generate server configurations and then finally Puppet, which I found (personally) to be simple to use/learn and didn't require me to directly learn Ruby unless I need to extend the 'language' of Puppet.
Theoretically you can use Chef without knowing Ruby, since if you're using pre-existing cookbooks you can just say what to install by listing the recipes in JSON format.
But practically speaking I'd say Ruby is essential. It'd be a disaster trying to use others' cookbooks without being able to read the recipes, and almost certainly you'll want to do something they don't cover.
On the other hand, perhaps Chef is an opportunity to learn Ruby? Certainly you can be productive with a very superficial understanding of the language, so you could get started pretty quickly. And a Chef recipe is about the flattest, simplest code imaginable.
Well, if you're using Chef and just using pre-existing cookbooks I'd say you take away 90% of the power of a CM system. So yeah, probably using Chef you have to know Ruby, full-stop.
For admins whom have a history with Ruby that's not a huge request, but at the same time I question how useful it is to basically require an admin to code in a particular language to build system configurations (And that is the one thing that drew me to Puppet... that while it had a 'language' it wasn't REALLY a language, more a template syntax).
I can certain agree with the idea of using Chef as an opportunity to learn Ruby and when the day comes I have some spare time to start then it might be a good place to begin using it.
Really though beyond the language aspect, I was wondering what the feature-set that Chef offered over Puppet, things like structural decisions like how nodes are managed, node removal (kind of a sore spot in Puppet right now), etc.
I don't know enough about Puppet to compare them, but the impression I got from reading several comparisons a year or two ago was that Puppet had a more mature community and hence better cookbooks, while Chef was better suited to "dynamic" responses, like adding nodes in response to high load. So perhaps Chef has better removal abilities.
Well, the community for Puppet is good, but oddly there are many modules (what Puppet calls a 'cookbook') missing for some relatively trivial things. For instance, I have written about 10 modules now for various system utilities/resources/services now which didn't exist as of yet, or at least were very... ahem... not good. For example, I've basically re-written my own nginx module because the ability to add reverse-proxy caching to server entries didn't exist even in the Puppet Labs nginx module (really, this is a thing)!
What's even worse is that the install script at https://www.opscode.com/chef/install.sh is also unavailable, which is the default way of getting chef installed onto newly provisioned server.
My overall experience with Chef isn't that great, because seems to me their team didn't response to users'issue seriously.
E.g. I have reported a bug [1] half year ago that the chef-server failed to start automatically when your server restart, still mark as `Unresolved` and no official answer from chef.
I think when your server startup script failed to work on a major distribution like Ubuntu, it should be considered as critical, right?
I'm sorry you feel that you didn't have a good experience.
As one of the original Chef contributors before Opscode, and then becoming one of the lead people working with the open source community for Opscode, the vitality of the open source Chef project is very important to me. However, Opscode's support for all of our open source projects also hinges on the company being successful enough to continue to do so. Thus, a balance must be struck.
Opscode doesn't respond to every ticket on that open source tracker, just like we don't write all of the code in the open source project. I personally try to at least see every issue, but with over 1.8 million installs of the Chef gem alone, there are a lot of users out there. If you want a guaranteed response, we do have various paid support plans which will provide you with that, but it doesn't come for free because we have to pay the people who research and write those responses.
There are multiple ways to install the Chef 10 Server and many ways to start the processes. We have tried to support the most popular, and I think most people have found success. We're proud that we can do things like release our rewrite of the Chef Server in erlang back to open source. We want it to be easy for people to use Chef. We initially wrote it to make our lives easier, and we want your lives to be easier too. That's why the Chef 11 Server ships in our Omnibus packaging, which provides a running Chef Server in three commands or less.
Firstly, thanks for your reply and your works on chef. I really appreciate it.
I think there is a difference between people who want free support & people who document a bug of a opensource project clearly in the hope of to improve the quality of that project - I hope I am in the latter case.
My reported issue can be easily fixed with some hacky workarounds, so I didn't bother with it and move on (As you can see I have never comment on that ticket). However, I still decided to create that ticket because I think the issue is easy to reproduce and will affect a lot of new comers to chef.
The issue is confirmed by others as well, one awesome commenter Florin Mihaila even provided a very detail analyse and solution to the issue.
For me, I would prefer if you can close the ticket if you think it is invalid, or mark as don't fix if resource is not allowed, rather than leaving as open and unanswered.
Of course, this is your project so I can't disagree with your preference, thanks anyway :)
Always appreciate bugs being filed. I did a little pondering on the ticket the other day. Since we're using update-rc.d, we're likely talking about debian package install, which means that we could probably set the defaults correctly with update-rc.d, someone just needs to do the research and patch the init scripts or the debian packaging. So it's best for the bug to be open.
Depressingly their competitors are no better. I've been monitoring a bug in puppet that's been open for the better part of three years for adding options like "--enablerepo" to yum invocations. This is not technically difficult to do but they keep dithering on design issues "because it's not cross platform". (Really? Because Ubuntu and RedHat totally agree on how to name packages, where system files live, which init.d to use, and those are both Linuxes; it gets even stupider with Solaris, BSD and Windows)
the combination of erlang backbone + ruby nodes is a wonderful way to write distributed systems. i've put in my time rolling my own when a single-node system needed to go distributed, and today i'd pay the up-front cost of integrating either erlang or a message queue as soon as i added the second node; these things invariably have to scale up more than you'd expect, and your homegrown system will almost as invariably fall over and die when you can least afford it.
Another thing I think helps to explain how beneficial a good configuration management system is, especially if you have a good deal of systems, is to think of the servers as part of a botnet, and chef as the command and control.
I started with Chef around version 10.4. It's really amazing to see how far Chef and the community have come since then. Keep up the awesome work Opscode!
I have even used Puppet but I could not get over the feeling that it is more complex than I would imagine for small end users (I have managed upto 40 servers with my libraries - it seemed quite feasible to double that. Maybe it's me but 100 servers sounds a lot and is manageable (deployment wise) with controlled scripts.