I once worked on an app where the staging database was used for local testing, all devs used the same shared credentials with write access, and you switched environments by changing hosts file entries (!!!). This resulted in me accidentally nuking the staging database during my first week on the job because I ran a dev script containing some DROPs from my corporate Windows system and failed to flush the DNS cache.
I had already called out how sub-optimal this entire setup was before the incident occurred but it rang hollow from then on since it sounded like me just trying to cover for my mistake. The footguns were only half-fixed by the time I ended up leaving some time later.
> I had already called out how sub-optimal this entire setup was before the incident occurred but it rang hollow from then on since it sounded like me just trying to cover for my mistake
That's what blameless post mortems are supposed to prevent. The only valid things to consider are questions like "is it a cost-effective prevention?", "what is the timeline of implementing this prevents / how should we prioritize?", and "does it meaningfully overlap with other changes that would prevent this?". Things like "well person A is new and that's why this happened" is not an acceptable position. Now of course, people can be not honest / not self-aware enough and are coming from that position without stating it anyway. That being said, that's why you try to focus on the objective evaluation of the recommendation and nothing else - that tends to make the bias irrelevant. Any EM/tech lead worth their salt would instantly say "is it actually important that staging and dev databases share creds"? Certainly prod should never. The other follow up to track down would be "if DNS flushing is so important when changing hosts, how have we automated it so that it happens automatically before every script that's run". I would also recommend to get rid of (ab)using the hosts file immediately and switch to explicit configuration files that you have to specify (perhaps automatically picking up "<username>.config" as a default so that scripts only ever run on your dev zone by default & locking down production.config so that only CI has the ability to touch that).
I absolutely concur with what you wrote but unfortunately not every team, company, or industry shares such a mindset. My teammates meant no offense and my mistake became a running joke but we never even considered doing an RCA for the problem. We did end up fixing the hosts file thing by moving to a different server-side architecture to rectify more pressing DX issues but the shared credentials issue persisted. One or two devs would actually lock the account by trying bad creds and block everybody's access - it took me diving into some RDBMS innards to figure out the responsible party and block them by hostname until they were able to correct the issue.
This is how most early stage companies (see: YC) are ran.
I’ve now taken 3 companies from horrid MVP (literally just got seed funding) to something a team enjoys working on. Each time I’m amazed that the app went without issues long enough to get funding (and long enough for me to refactor the whole thing.)
This is how I accidentally had a test message sent as a push notification to a few ten thousand people; our test/dev mongodb instance was on the same machine as the production instance, the boss asked me "can you switch it to the test db so we can send a test message", I was like "sure ok it's on test now" but it was not. Whoops.
Man, that's such a bizarre idea...that's gotta be just about the worst example of hosts file abuse I can think of. But then again, I'm almost afraid of what other examples posters here might share, lol
I had already called out how sub-optimal this entire setup was before the incident occurred but it rang hollow from then on since it sounded like me just trying to cover for my mistake. The footguns were only half-fixed by the time I ended up leaving some time later.