I find people dont know how amazing a "immutable" server fleet is until you've experienced it.
It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour.
It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.
"It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour."
Got any tips in this regard?
Ansible, Puppet, Python, Terraform, Openstack Pulumi springs to my mind. Still need to cover a PXE server. (Maybe with Ansible)
We use Salt Masterless with Packer to build AMIs.Terraform defines our infra. No matter what happens we ensure any service can boot by itself with 0 intervention
It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour.
It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.