Usually it starts with a company having issues with robot traffic. So they try a bunch of things to hinder the robot(s). They do something, the robot stops working, but after a while it comes back, it's a cat and mouse game essentially.
One day, they (developers pushed by middle managers) disable copy-paste on the login page, and the robot temporary stops working, until a couple of days later, when the robot found a way around it.
On to the next thing to do to stop the robot, but that previous "fix" is still there, with the thinking that "maybe that stops some of the robots", but it probably doesn't.
But there it sits, some ~10-ish lines of JS that will hang around until rewrite v6 when they'll begin from the beginning, and some months/years later come around to disabling it once again.
You can't win; you're going to get robot traffic unless everybody does something like Web Environment Integrity. Seriously.
Just allocate your finite resources in a hierarchical 32-level binary tree based on bit prefixes of the client IP address. Exactly what the root DNS servers do. And exactly what the only mitigation for slowloris attacks does. Then get on with your life.
Honestly, robot traffic seems like an issue just because websites are horrendously inefficient. Hundreds of round-trips to external servers, languages that require some orders of magnitude more resources than needed... It shouldn't be so expensive to just serve a page, even considering robots
> Hundreds of round-trips to external servers, languages that require some orders of magnitude more resources than needed... It shouldn't be so expensive to just serve a page, even considering robots
This is not a priority. The features are implemented by more abstraction, ie. TypeScript and web frameworks. Industry's low barrier to entry promotes studying frameworks, not technologies and standards enabling them. Anti-robot measures mostly prevent automated fraud and are there to ensure the ads are displayed, if the whole process will freeze your browser and eat your entire RAM they are fine with it.
You’re right on that part but I think it isn’t so much the server resources but the actual things the bots may be doing. For example making a ton of bot accounts to spread propaganda, or 10,000 “trial accounts” to host untraceable phishing/scam pages, etc. Or for example, an e-commerce site that doesn’t want to be automated into service as a card tester for stolen credit cards with thousands of fraudulent orders.
Idk what you are saying, are you suggesting if I operate a webstore I should let bots place thousands of fraud orders frequently, and eat all those chargeback fees? And… law enforcement? Call the cops every time that happens? At least in my country, the police would say “uhh, ok feel free to file a report,” but they will do zero to investigate it. Which actually makes sense since most of those doing this crime are operating overseas, out of their jurisdiction anyway.
Also, if someone is registering 10,000 accounts that are obviously not real people, I should let them?
First of all, my website, my free speech. I’m free to publish or delete anything on it.
Second, bulk-created fake accounts aren’t needed even for legitimate political speech. That’s more like extreme astroturfing.
That's exactly what developers will tell middle-managers but it won't matter unless you're in a organization that actually value their developer's opinion.
One day, they (developers pushed by middle managers) disable copy-paste on the login page, and the robot temporary stops working, until a couple of days later, when the robot found a way around it.
On to the next thing to do to stop the robot, but that previous "fix" is still there, with the thinking that "maybe that stops some of the robots", but it probably doesn't.
But there it sits, some ~10-ish lines of JS that will hang around until rewrite v6 when they'll begin from the beginning, and some months/years later come around to disabling it once again.
No, I'm absolutely not speaking from experience.