> We have to crawl about 60-80k news websites per day [0] Can't even imagine tha... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

jcun4128 on Sept 13, 2021 | parent | context | favorite | on: How to Crawl the Web with Scrapy

> We have to crawl about 60-80k news websites per day [0]

Can't even imagine that number... different languages or something?

artembugara on Sept 13, 2021 [–]

Yeah, plus many of websites are quite niche news (construction news, for example)

Join us for AI Startup School this June 16-17 in San Francisco!
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact