I mean the main reason is that robots.txt is pointless these days.
When it was introduced, the web was largely collaborative project within the academic realm. A system based on the honor system worked for the most part.
These days the web is adversarial through and through. A robots.txt file seems like an anachronistic, almost quaint museum piece, reminding us of what once was, while we stoop head first into tech feudalism.
When it was introduced, the web was largely collaborative project within the academic realm. A system based on the honor system worked for the most part.
These days the web is adversarial through and through. A robots.txt file seems like an anachronistic, almost quaint museum piece, reminding us of what once was, while we stoop head first into tech feudalism.