Old hat here. Things certainly can get worse. Outages can turn into a loss of customer data. Attempts at restoring lost data can result in corrupting an unknown amount of customer data.

Throw caution to the wind and you can turn a 20-minute prod outage into a multi-day all hands disaster recovery. It's best to get your heart rate down, put on your captain's hat, and choose your next step carefully.

Yeah that why my job has incident managers. When you get paged you only invesitgate. No touchy. When the team bands together, facts are shared and decisions are made carefully. Usually gonna be stuff like roll backs.

I think old saying “slow is quick, quick is slow” applies here. Don’t do anything stupid because you are in panic mode.

Slow is smooth and smooth is fast

The Junior hat, absolving you from any responsibility for prod's status.

Not sure I got you right here but this is exactly the situation where you should calm your breathing and make triple sure you don't make things worse -- because things can always get worse.

The red hat.

aka The Pirate Hat

macgruber hat