Hacker News new | past | comments | ask | show | jobs | submit login

I'm genuinely interested in what the infrastructure was that couldn't support orchestration of Puppet clients. I hear this from people sometimes and it usually ends up being related to poorly architected Puppet infrastructure for their environment.

A properly architected Puppet environment should have no problems dealing with thousands of clients.




Depends what you want to do. If you're happy for changes to dribble in over time then size your puppetmaster pool to how many hosts you want to be able to run puppet simultaneously and stagger client execution to avoid a stampeding herd. Accept that sometimes there will be individual failures due to load and clients will just have to wait till the next time.

Alternatively, in real life, many teams have Change Management to consider and Maintenance windows. If there's a need to update thousands of systems on a saturday morning then expect teams to start puppet runs manually. You'd better have a seriously big pool of puppetmasters ready and waiting to manage the load, and don't forget Puppet DB, that has to be scaled up too to avoid lock ups. Even then, if teams start too many puppet runs at once, you'll get flattened.

We ended up scrapping all the puppetmasters in individual DCs and consolidating them in an AWS EC2 Autoscaling group. The number of puppetmasters started at 70 and just went up. That came with problems of its own. e.g. ensuring that all puppetmasters share the same copy of role versions at the same time. Being able to spin up new puppetmasters fast enough to meet spikes in demand. Various other corner case tuning issues.

It's taken a dedicated team years to get to grips with puppet, tame it and master it. Very glad I'm not involved in that any more.


The non-deterministic run order was a nightmare to debug so everyone moved to ansible


That's exactly it. Trying to fight to get things to run in a proper order was a nightmare.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: