Every website operator should be required to learn _rate-limiting_ as well as how to create a robots.txt file

What matters is rate not "user-agent"

User-agent can be fake but rate is real