Interesting to see that this is kicked off by the referal header. Seems like a privacy issue to have your browser tell servers part of your browsing history by default.
Interestingly enough, it's configurable both from the user side and from the referring site side.
Most (all? all the relevant ones, I think) browsers honor the referer-policy[1] header if a referring site sets it. There are options in common site frameworks, like Django[2] to control that for UAs that respect it.
Since most UAs respect it, if the indexing site had wanted to, they could easily have prevented the header from being sent for most users.
IIRC, Tor doesn't have that issue.