OK, so my most memorable moment was The On Call Day From Hell.
I've told this story a few times a few different ways, but for this purpose, you could call what we did theatre. Every 5, or 10, or 15 minutes, all weekend, some weekdays and evenings, the show must go on. ~6 Locations spread through the city, two separate experiences, both with their own windows, problems, and margins for error.
Sunday morning comes around and I get an alert about one of the locations. The system didn't come online (startup was automated for one experience, and manual, but staffed for another). The automated experience wasn't booting up. I had a three hour window to make the trek from Oakland to SF and debug this, probably minor issue.
I had VPNs everywhere, if I needed to show up, it was usually a hardware or networking issue. I was grumbling as I left my girlfriend on Sunday morning. I felt like it wasn't my turn (there were two other engineers, one other who would sort of rotate these duties at the time). So I'm trudging along and I get a text from the staff member running the other thing. She tells me that "there's water under the tunnel". This is very bad. This particular thing was built into a basement, which was at one time a creek. Even in the drought we were pumping ...water (and mission st run off)... out fairly constantly.
So I parlay the problem downtown by forcing an unexpected reboot of all equipment and make sure it sets itself up properly. I move to the mission and I'm woefully underprepared. The set itself has begun getting wet, I go to the source and it's just a black hole. Our construction crew set it all up initially, and would normally be the go-to for this, but we were a very small crew. Construction wasn't responding, and the first appointment was in 45 minutes. This was my problem. So I psych myself up. I was going to something else and definitely not planning on getting shoulder deep in sludge. I undress as much as I can around other employees, the two of us cover the set with as much protective barrier as we can and I just stare at it.
I'm about to get into the sort of territory where I'm cautious about telling it to the HN audience, but I shit you not, a voice from inside the hole, and from a very good friend whispers to me "psssh. Hey, you, It's down here. Come on down".[1]
And I get shoulder deep in a sump pump field disassembly and repair. No appointments were missed, edited or canceled that day.
[1] There's a very practical and boring explanation for this, but it was a very laugh-out-loud at my own weird life sort of moment.
I've told this story a few times a few different ways, but for this purpose, you could call what we did theatre. Every 5, or 10, or 15 minutes, all weekend, some weekdays and evenings, the show must go on. ~6 Locations spread through the city, two separate experiences, both with their own windows, problems, and margins for error.
Sunday morning comes around and I get an alert about one of the locations. The system didn't come online (startup was automated for one experience, and manual, but staffed for another). The automated experience wasn't booting up. I had a three hour window to make the trek from Oakland to SF and debug this, probably minor issue.
I had VPNs everywhere, if I needed to show up, it was usually a hardware or networking issue. I was grumbling as I left my girlfriend on Sunday morning. I felt like it wasn't my turn (there were two other engineers, one other who would sort of rotate these duties at the time). So I'm trudging along and I get a text from the staff member running the other thing. She tells me that "there's water under the tunnel". This is very bad. This particular thing was built into a basement, which was at one time a creek. Even in the drought we were pumping ...water (and mission st run off)... out fairly constantly.
So I parlay the problem downtown by forcing an unexpected reboot of all equipment and make sure it sets itself up properly. I move to the mission and I'm woefully underprepared. The set itself has begun getting wet, I go to the source and it's just a black hole. Our construction crew set it all up initially, and would normally be the go-to for this, but we were a very small crew. Construction wasn't responding, and the first appointment was in 45 minutes. This was my problem. So I psych myself up. I was going to something else and definitely not planning on getting shoulder deep in sludge. I undress as much as I can around other employees, the two of us cover the set with as much protective barrier as we can and I just stare at it.
I'm about to get into the sort of territory where I'm cautious about telling it to the HN audience, but I shit you not, a voice from inside the hole, and from a very good friend whispers to me "psssh. Hey, you, It's down here. Come on down".[1]
And I get shoulder deep in a sump pump field disassembly and repair. No appointments were missed, edited or canceled that day.
[1] There's a very practical and boring explanation for this, but it was a very laugh-out-loud at my own weird life sort of moment.