I have a server that has a slow (5s) response to unknown pages, returns it as 200, and makes the next failing request even slower (for unauthenticated users). That seems to keep the number of requests limited. Perhaps I should just drop the connection after a certain number of requests.

BTW, quite a few of these port scanners are companies that offer to scan your ports for vulnerabilities. Temu pen testing, so to speak.

Do you configure this in your firewall? How can I replicate this?

what firewall do you use?

It's in the "404" handler of the backend. It should be possible to write a caddy or nginx module for it.