Could be because you’re using a terrible ISP-provided router. Switch that to bridge mode and connect the server directly (if necessary, use the server as a router with a second network card for your LAN) or use a proper, enterprise-grade router (OpenWRT would work as well as long as the underlying hardware is performant enough).
Sadly, not possible on a shared residential connection. I tried the setup where the server was also the router, and that was much less convenient in case of failures than having a dedicated router.
But yes, the abundance of routers is definitely a problem here, with an OpenWRT and a FritzBox in the mix. Perhaps I should try to get IPv6 running and forget about NAT woes.
I think it might have something to do with the NAT forgetting about my UDP "connection", but haven't found the culprit yet.