Wikia Buys Grub from LookSmart

Wikia has purchased the Grub distributed web crawler from LookSmart.

Grub is like seti@home for web crawling. It enlists idle computers to crawl and analyze the web for signs of life. Just kidding… It does however allow you to donate your idle computer and network connection towards a world-wide effort to crawl the web in search for new and changed content.

From Grub.org:

Grub is back. Combining revolutionary crawling technology, the power of distributed computing, and a global community of volunteers, Grub seeks to crawl every website, every day and to provide definitive answers about the true nature of the Web.

As the structure of the Web changes day by day, Grub will document its evolution. It will also provide the data for peerless search results as new content is crawled and uncharted sites are discovered.

Currently back from the afterlife, Grub will be ramping up its innovative crawling model and testing integration with search once again. Every person who donates their computer’s unused bandwidth speeds up the project and helps to realize the goal of charting the entire Internet.

Our t-shirt slogan for Grub used to be “Find out where the Internet Ends”.

I was particularly interested aquiring Grub when I was CTO of LookSmart because a distributed crawling model holds the promise of being able to check web pages to see if they have changed much more often than the centralized web crawling effort that Google/Yahoo/MSN each employs.

This is a critical issue in web search for all sorts of reasons as most people do not realize that the search results they see in the major engines are reflective of pages that existed on the web 30 days ago.

Now, some of those documents are updated more frequently, but the issue for the web search engine is knowing when a page on the web has changed. The reality is that you don’t know if a web page has changed unless you go an check it. That’s where Grub comes in by providing a massive coordinated amount of computing power and network bandwidth needed to check every page on the internet as often as every day…

I’m glad to see focus return to Grub through Wikia and its open source roots live on. It truly does have the potential to change the way that web search is done.