I don't think that this is the answer to "only google can crawl the web". This is a huge archive suitable for making a web search engine maybe.
What if you want to make a simple link previewer? An abstract crawler for scientific articles? Most websites are behind cloudflare which will block/captcha you, but happily whitelist only google & major social sites. Tha answer is measures that bring the web back to basics, not this over-SEOed bot infested ecosystem. FANGS succeeded in sucking out all the information of the web, but they suck at creating protocols that are interoperable (even twitter now needs its own tags!).
Incidentally, maybe the next search engine should use a push-system , where websites will ping to it whenever they have updates. If the engine has unique features, it might be actually seen as a measure to reduce the loads from bots.
Actually I was able to search URLs with very little ram across the entire collection as they have a series of indexes you can download.
In theory someone could do something similar with terms, or you could first use URLs to filter the text size you download into Elastic or Solr and do your own custom search that way.
The indexes are really neat though, I highly recommend playing with them.
I recently did a pass through CC it took me 5 days, there were 80,000 1GB warc files. I had to just filter the information I wanted and dump the rest because I can't possibly host that much data on my machines. It's great data but very noisy, like the web is. Lots of repetition and useless junk mixed with useful data.
>maybe the next search engine should use a push-system
So setting up a new search engine that way would require going to every site and convincing them to notify you of changes. Wouldn't that be even more limited than the current Cloudflare whitelist system? At least there's some chance you can get around the whitelist system.
This... I couldn't get past the irony of the comment. Basically, the problem is that sites only let google and friends index them. The solution? Site should only send index data to google and friends.
I mean, I get it, then sites can send to any number of indexers, but let's be honest, like you say, any new search engine has to get sites to push data out to them. That's just not. going. to. happen.
I don't think we can bring the web back to basics in the sense you're envisioning without kicking most users off of it.
Cloudflare's protection is to guard against traffic spikes and automated malicious attacks; Google and social sites are allow-listed because they're trustable entities with well-defined online presences and a vested interest (generally) in not breaking sites. "How do we do away with the need to put up an anti-automated-traffic screen plus whitelist" is in the same category of problem as "How do we change the modern web to address automated malicious attacks?"
Another way to look at this is if CDNs don't allow Google, nobody is going to want to use them. Their content doesn't get indexed and anybody doing a search is going to get directed to someone else that doesn't put their content behind a CDN with that level of protection. That or someone like google will just solve the problem themselves and be both the CDN and the indexer, bringing them one step closer to complete ownership of finding anything on the web.
> Cloudflare's protection is to guard against traffic spikes and automated malicious attacks
This simply isn't true. By my reading, Cloudflare implies any automated traffic that doesn't openly advertise itself as such is bad. They certainly seem to be happy to assist you in blocking it regardless of whether it was actually malicious or actually causing traffic problems. (https://blog.cloudflare.com/super-bot-fight-mode/)
I'm not sure I understand the distinction you're making. Automated traffic that isn't flagging itself as automated is deceptive, and that's a negative signal for maliciousness right out of the starting gate.
Shouldn't automated traffic be allowed privacy? To not do so means we have to take away privacy from humans in order to verify they are not automated. What do we gain for this? Why shouldn't some programmable bot be allowed to see my content but any human no matter their intent are allowed by default?
Good resource, admirable intention, great that it simply exists. Good sized index.
I see a lot of people subscribe to the idea of this being the feeder to alternative search engines.
I'd guess part of the problem with doing things this way is the 'crawl priority' of what the search engine thinks are the next best pages to crawl, it's totally out of their hands or at least, they'd still need to crawl on top of the Common Crawl data.
The recent UK CMA report into monopolies in online advertising estimated Google's index to be around 500-600 billion pages in size and Bing's to be 100-200 billion pages in size [0]. Of course, what you define as a 'page' is subjective given URL canonicals and page similarity.
At the very least, the common crawl gets around crawl rate limiting problems by being one massive download.
Would be interesting to know if there's an appreciable % of site owners blocking it, though going on past data (there is some data in the UK CMA about this also), it's not a huge issue.
As an aside, it always jars me when a site hijacks default browser scrolling functionality. In my experience, making it as fast as possible is a _far_ better use of dev resources than figuring out how to make scrolling unique (no matter what the marketing department says).
I've used this and it's invaluable for all types of things but a feeder for Google killers it is not.
They don't approach the scale of what Google crawls, they state as much. Nor do they do it on the same timeline as Google. This is really nice for research or kick starting a project but this isn't a long term viable solution for alternative search engines. Between breadth, depth, timeline/speed, priority, and information captured it falls well short.
Well that's to be expected, but if a lot of search engines start using it, it's likely that websites will start allowing it to crawl and index their pages. So there might be potential there.
How feasible would it be to store all that data on a decentralized system like IPFS or Sia-Skynet etc, instead of Amazon, to add further meaning to the cause?
You may not grasp just how large the Common Crawl dataset is. It's been growing steadily at 200-300 TB per month for the last few years. I'm not certain how large the entire corpus is at this point, but it's almost certainly in the tens to low hundreds of petabytes. (This is significantly larger than the capacity of the entire Sia network, for example.)
Storing a dataset of this size and making it available online is not inexpensive. Amazon has generously donated their services to handle both of these tasks; it would be foolish to turn them down.
It's not clear, but it looks like the last crawl was 280 TiB (100 TiB compressed) and contains a snapshot of the web at that point; i.e. you don't need prior snapshots unless you're interested in historical content.
As best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months):
In fact someone could easily setup some IPFS nodes that fetch the data from the current host if requested over IPFS.[1] This way people could access it via IPFS and provide an alternate mirror of the data.
- Even if the source is unavailable there may be other copies on IPFS which would be transparently used.
- There may be some performance benefits in rare cases.
- If you are accessing this on a bunch of machines your IPFS gateways would handle downloading the source once, then automatically using the local copy from inside your network.
The maindownside is that if Amazon is donating their resources why bother with IPFS?
That's not correct. When it comes to storage and transfer, blockchain alternatives are fraction of the cost of Amazon. For example Sia Skynet is offering $5/month/TB[1] storage. If you skip Skynet and run your own Sia node the price can even go lower to $2/month/TB basing on the market conditions.
It seems that at least on Sia's plans, you can maximally host 20TB for 80$/month, not even a tenth of a monthly common crawl.
Of course Sia's Skynet are package deals right now and I guess they're currently bootstrapping the network with users. Filecoin has no operational storage yet. Storj quotes 10$/Terabyte/month [1] so that will come out expensive.
Moreso than Amazon? From my (limited) experience blockchain storage solutions are often less expensive, although I've never worked with petabytes of data so maybe it's different on that scale.
As a European (German) I am always wondering about the legal basis for a) making a copy of copyrighted material and databases available and b) processing contained personal data.
The Internet archive seems more like a library with exceptions applying, but common crawl seems to advertise also many other purposes that go beyond archiving publically relevant content.
Would this be possible in Europe, too? My feeling is that US legislation different here. Do you have to actively claim copyright in the US or enforce technically e.g. via DRM? Anyone can use anything without a license as long nobody finds out?
Well they've got a bunch of different data that would fall under different precedents. Metadata would likely fall under the phone book ruling. The raw warc data could be argued to be transformative, and would certainly fall under fair use - especially because it isn't even close to threatening market replacement. Property abandonment might fit in somewhere as well :)
I am curious about this too, from a copyright perspective:
1) Whether it is legal for Common Crawl to collect this data in the first place? Presumably they don't have explicit permission from every site they have crawled?
and
2) Whether it is legal to use this data to train ML algorithms for commercial use (e.g. GPT-3)? They provide a bunch of examples of this on their site but the terms of use are kind of vague about this.
It has to be legal to use legally acquired published information for a commercial purpose. That is not nearly the same thing as selling someone else's work. Training a ML algorithm on published work is no different from me reading a tutorial on the web and using the new skills I learned to do something that makes me money. That is very different from reselling the tutorial as if I authored it.
The data is open they don't crawl inside password protected areas of the internet. Why shouldn't they be able to read data put into the open? If the authors didn't want their data openly accessible they should put it in a private place.
It seems like Common Crawl is doing a lot of awesome stuff, but they're not attacking Google's stranglehold head on.
Presumably this is because they lack the money to do so. Have they attempted to estimate how much it would cost per year to crawl the web as aggressively and comprehensively as Google does? I've checked their site and didn't find anything like that.
If they came up with a number, say $2 or $10 billion per year, it might actually be possible to gather enough donations to dethrone Google.
A lot of Google competitors would love to see them dethroned. And it would be a huge win for virtually everyone else too. There's no one in the world that wants Google to maintain their web search monopoly indefinitely.
It'll be new to some people every time it's posted because new people are entering the industry or signing up with HN or developing new interests every day. That's one reason I personally don't mind reposts or old content.
What if you want to make a simple link previewer? An abstract crawler for scientific articles? Most websites are behind cloudflare which will block/captcha you, but happily whitelist only google & major social sites. Tha answer is measures that bring the web back to basics, not this over-SEOed bot infested ecosystem. FANGS succeeded in sucking out all the information of the web, but they suck at creating protocols that are interoperable (even twitter now needs its own tags!).
Incidentally, maybe the next search engine should use a push-system , where websites will ping to it whenever they have updates. If the engine has unique features, it might be actually seen as a measure to reduce the loads from bots.