I have a server that has a slow (5s) response to unknown pages, returns it as 200, and makes the next failing request even slower (for unauthenticated users). That seems to keep the number of requests limited. Perhaps I should just drop the connection after a certain number of requests.

BTW, quite a few of these port scanners are companies that offer to scan your ports for vulnerabilities. Temu pen testing, so to speak.