And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right?
Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door.
> And of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right?
If you don't spend the time writing a super-duper BGP implementation using sexy Rust rather than dealing with boring Cisco and Juniper boxes supporting all kinds of orchestration that gets you onto front page of Hacker News, you can get a set of Expect scripts running against serial consoles 24x7 triggering notifications of OOB failures within minutes.
Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door.
(again, not referring to today's outage)