I blame the wifi. I can ssh from one wired machine to another just fine, but when I unplug the cable the wifi doesn't seamlessly take over. This is clearly the fault of the wireless.
I think it's a problem with the TCP/IP stack somehow. If I try the same thing when downloading something from chrome, the download fails. If I'm loading a website I get a "connection interrupted" error. Really there's no reason that my device shouldn't have its own IP and treat wifi and ethernet as two different routes to my device. Remember "IP routes around damage"?
Nothing to do with Wi-Fi, the same would happen if you had 2 Ethernet connections.
Your Wi-Fi and Ethernet are both network interfaces with their own IP addresses - those addresses don't move when you switch between interfaces. If you established an SSH session while on Ethernet and then unplugged it, that session will not carry over to Wi-Fi unless you change your Wi-Fi interface's IP address to the wired one's and that both networks are on the same L2 segment (typical for home networks, but not always the case for enterprise).
A potential solution is either MPTCP (which will establish multiple connections over all interfaces and can tolerate the loss of all but one of them) or to VPN into your router and use the VPN link as your default route - this means that the VPN connection might drop and reconnect but your IP address to the other hosts (which is actually the one of your VPN gateway) will remain constant, so traffic flow will be restored as soon as your VPN reconnects even if your local IP changes.
If you configure things so that the wifi adapter gets the same IP address as the wired connection you should be able to continue communicating with those machines. SSH uses TCP which is connection-oriented so as long as the connection can be made it should work (bar problems with MTU etc.). You might experience a stall before the connection becomes alive again but it should work.
Sun/Solaris machines used to have a machine specific MAC address. Then DHCP, bootp etc would give it the same IP independent of interface. Wonder what would happen in a current *BSD/Linux system if you gave switched the MAC address over.
It's been literal decades since I configured this (and carried it forward with the rest of my dotfiles), but there's a configuration setting that makes SSH reconnections of existing sessions seamless. No, I don't know what it is off the top of my head, let's call it an exercise left to the reader.