Categories: Tech & Society

Common Crawl – Free Database Of The Entire Web, Competition For Google

We all know Google! Google started off as nothing but a website indexer, built a more efficient algorithm for ranking Web pages and finally built its success on crawling the Web—using software that visits every page in order to build up a vast index of online content.

A nonprofit called Common Crawl is now using its own Web crawler and making a giant copy of the Web that it makes accessible to anyone. Common Crawl supplies a database of over five billion Web pages that can be accessed and analyzed by anyone, in the hope that it will inspire new research or online services.

“The Web represents, as far as I know, the largest accumulation of knowledge, and there’s so much you can build on top,” says entrepreneur Gilad Elbaz, who founded Common Crawl. “But simply doing the huge amount of work that’s necessary to get at all that information is a large blocker; few organizations … have had the resources to do that.”

Elbaz says he noticed around five years ago that researchers with new ideas about how to use Web data felt compelled to take jobs at Google because it was the only place they could test those ideas. He says Common Crawl’s data will make it easier for novel ideas to gain traction, both in the world of startups and in academic research.

Common Crawl has so far indexed more than five billion pages, adding up to 81 terabytes of data, made available through Amazon’s cloud computing service. For about $25 a programmer could set up an account with Amazon and get to work crunching Common Crawl data, says Lisa Green, Common Crawl’s director.

More details at MIT Technology Review.

Prateek Panda

Prateek is the Founder of TheTechPanda. He's passionate about technology startups and entrepreneurship and enjoys speaking to new founders every day. Prateek has also been consistently regarded as one of the top marketing experts in the region.

Recent Posts

Remote working is bringing about a leadership evolution

It was COVID-19 that truly heralded in remote working culture, and ever since, workdays have…

14 hours ago

As US restrictions blow the chip industry where will the chips fall?

As tensions between global superpowers escalate, semiconductors, those tiny chips powering everything from smartphones to…

3 days ago

AI Launches: Fashion, EdTech, speech tech, monitors, fintech, data, trading, cloud, AI defense & model

The Tech Panda takes a look at recent launches in the superfast field of Artificial…

4 days ago

6 Automation tools every 2D animator should master in 2025

As the animation industry continues to evolve, automation is becoming a critical ally for 2D…

4 days ago

Can you really find parking in 5 minutes? Tech promises a breakthrough in Delhi-NCR

If you’ve ever circled endlessly in Delhi-NCR’s crowded streets, searching for a parking spot while…

7 days ago