It's a short blog post that I've probably read a thousand times at this point. He wrote about how websites give Google a big advantage when it comes to web crawling and how that big advantage probably makes it harder for other search engines to compete with Google. This was a pretty striking idea to me and there was a lot of talk at the time about antitrust and Big Tech. Dan's post had been written in 2015, so I was sure that a ton of other people, especially DC policy people, already knew about this and were talking about it. Right?
Turns out, basically nobody in DC knew anything about this. A ton of website operators complain about it on their own forums like HN and SEO, it's not hard to find people griping about the cost of Bing's crawlers, but those people never saw fit to tell anybody in DC about this and how it impacts the market for general purpose search engines and gives Google such an advantage. So I started writing down everything that I was finding about Google's web crawling advantage and writing it in a way that policy people could understand these things called web crawlers they had probably never heard much about before.
And, long story short, the policy people were very grateful that I had gotten in touch and explained all this, and I got cited in the Big Tech Antitrust report published by Congress last summer and then featured in The New York Times:
So, Dan's blog post has had a pretty big impact on my life so far and it's not quite done yet. The pandemic has slowed me down this year much more than it did last year, but I'm working on preparing to submit a paper to an economics journal for peer review that lays out the dynamics of web crawling, why Google accrues this advantage and why it matters. I'm very grateful to Dan for writing that post and, as far as career advice goes, I heartily recommend going back every once in a while and rereading everything he has ever written. Who knows what else he's hiding in there?
The difference between Bing and Google's crawlers causes me physical pain.
As part of a team that runs several large sites, the performance difference between them is stunning, but not any any way complimentary to Bing. We want Bing to do better. We want them to challenges Google pole position.
Bingbot will crawl the exact same set pages 10 times in a week, and will make hundreds of requests per minute doing so. Googlebot will crawl them once, at a much more reasonable pace.
The bandwidth and power costs we incur from Bing crawlers are a hundredfold (at least) what we incur from Googlebot.
Please, Microsoft, make Bingbots not suck so badly.
https://danluu.com/googlebot-monopoly/
It's a short blog post that I've probably read a thousand times at this point. He wrote about how websites give Google a big advantage when it comes to web crawling and how that big advantage probably makes it harder for other search engines to compete with Google. This was a pretty striking idea to me and there was a lot of talk at the time about antitrust and Big Tech. Dan's post had been written in 2015, so I was sure that a ton of other people, especially DC policy people, already knew about this and were talking about it. Right?
Turns out, basically nobody in DC knew anything about this. A ton of website operators complain about it on their own forums like HN and SEO, it's not hard to find people griping about the cost of Bing's crawlers, but those people never saw fit to tell anybody in DC about this and how it impacts the market for general purpose search engines and gives Google such an advantage. So I started writing down everything that I was finding about Google's web crawling advantage and writing it in a way that policy people could understand these things called web crawlers they had probably never heard much about before.
And, long story short, the policy people were very grateful that I had gotten in touch and explained all this, and I got cited in the Big Tech Antitrust report published by Congress last summer and then featured in The New York Times:
https://knuckleheads.club/we-crawled-our-way-into-the-big-te...
https://www.nytimes.com/2020/12/14/technology/how-google-dom...
So, Dan's blog post has had a pretty big impact on my life so far and it's not quite done yet. The pandemic has slowed me down this year much more than it did last year, but I'm working on preparing to submit a paper to an economics journal for peer review that lays out the dynamics of web crawling, why Google accrues this advantage and why it matters. I'm very grateful to Dan for writing that post and, as far as career advice goes, I heartily recommend going back every once in a while and rereading everything he has ever written. Who knows what else he's hiding in there?