Categories: Tech & Society

Common Crawl – Free Database Of The Entire Web, Competition For Google

We all know Google! Google started off as nothing but a website indexer, built a more efficient algorithm for ranking Web pages and finally built its success on crawling the Web—using software that visits every page in order to build up a vast index of online content.

A nonprofit called Common Crawl is now using its own Web crawler and making a giant copy of the Web that it makes accessible to anyone. Common Crawl supplies a database of over five billion Web pages that can be accessed and analyzed by anyone, in the hope that it will inspire new research or online services.

“The Web represents, as far as I know, the largest accumulation of knowledge, and there’s so much you can build on top,” says entrepreneur Gilad Elbaz, who founded Common Crawl. “But simply doing the huge amount of work that’s necessary to get at all that information is a large blocker; few organizations … have had the resources to do that.”

Elbaz says he noticed around five years ago that researchers with new ideas about how to use Web data felt compelled to take jobs at Google because it was the only place they could test those ideas. He says Common Crawl’s data will make it easier for novel ideas to gain traction, both in the world of startups and in academic research.

Common Crawl has so far indexed more than five billion pages, adding up to 81 terabytes of data, made available through Amazon’s cloud computing service. For about $25 a programmer could set up an account with Amazon and get to work crunching Common Crawl data, says Lisa Green, Common Crawl’s director.

More details at MIT Technology Review.

Prateek Panda

Prateek is the Founder of TheTechPanda. He's passionate about technology startups and entrepreneurship and enjoys speaking to new founders every day. Prateek has also been consistently regarded as one of the top marketing experts in the region.

Recent Posts

Why everything is a subscription now & what it means for your wallet

In the last decade, purchasing and accessing products and services have undergone a radical transformation.…

20 hours ago

Mumbai students win big at robotics championship in Chicago, headed to world finals

In an exceptional achievement, Mumbai-based high school robotics team Sigma 9692 has won the prestigious…

23 hours ago

Indian gaming shifts from audience to creator “given India’s mobile-first gaming capabilities”

The Indian gaming industry is sitting on an industry explosion in the near future as…

2 days ago

Transforming modern business operations: How Microsoft 365’s AI Integration is automating HR workflows & enhancing employee experiences

In today's fast-paced business world, when employee fulfillment and efficiency are crucial, technology has become…

2 days ago

Outbound & inbound: India attracts businesses from UK, China & US while expanding to Middle East

The Tech Panda takes a look at how India has been attracting foreign businesses from…

3 days ago

UPI value & volume surpass records in March

India’s digital payments ecosystem achieved a record high in March, with UPI transactions reaching INR24.77…

3 days ago