At the urging of the tech lead at the time, my team has pursued the ultimate ssh trick: not using it.
i.e. we intentionally do not enable (human) ssh access to the production hosts. In an autoscaling AWS world, logging into an individual machine by hand is the last thing you want to be doing. So we are learning the (sometime difficult!) lessons of how to rely only on what our logging, tracing, monitoring, and deployment automation (including snapshotting) can afford us. I suspect sooner or later we will break down and swap in a login-enabled image to diagnose some sticky problem, but -- as much as I resented the idea when he presented it -- it's an interesting discipline.
This is great, I think at first glance it works at scale, but I think a lot of shops not in the 'startup world', older IT shops still working out of the 90s and 2000's will find that hard to do. When you have 5 monster work horse servers instead of 50 VMs where you can spin up 50 more within 30 minutes it's hard to get away from direct server access for the devops team.
I like the idea of it, given how expendable servers are.
I'd love to hear any instances where you have SSH'ed in, or what kind of bugs would prompt you to want to.
Maybe because you didn't have some monitoring on a specific aspect of the server, or an application bug where you needed to use strace - something like that?
This is the same principle I have started to use with Docker. It runs a single application, stuff logs, but besides that there is no way to get into it, no ssh, nothing.
Lot of places disable it for the production hours of operation and enable it for maintenance windows. Done using access control not via disabling it though
i.e. we intentionally do not enable (human) ssh access to the production hosts. In an autoscaling AWS world, logging into an individual machine by hand is the last thing you want to be doing. So we are learning the (sometime difficult!) lessons of how to rely only on what our logging, tracing, monitoring, and deployment automation (including snapshotting) can afford us. I suspect sooner or later we will break down and swap in a login-enabled image to diagnose some sticky problem, but -- as much as I resented the idea when he presented it -- it's an interesting discipline.
Anyone else living by that principle?