{"id":1013,"date":"2007-11-07T01:42:32","date_gmt":"2007-11-06T23:42:32","guid":{"rendered":"https:\/\/english.martinvarsavsky.net\/general\/pinging-vs-crawling-and-an-open-source-search-engine.html"},"modified":"2007-11-07T01:48:49","modified_gmt":"2007-11-06T23:48:49","slug":"pinging-vs-crawling-and-an-open-source-search-engine","status":"publish","type":"post","link":"https:\/\/english.martinvarsavsky.net\/?p=1013","title":{"rendered":"Pinging vs Crawling and an open source search engine"},"content":{"rendered":"

I am sure that there\u00b4s tons of stuff written on the web about the pros and cons of pinging (notifications a la technorati) vs crawling (programs that scout the web for links a la google) or listening vs spying. Tonight we had dinner with Jimmy Wales<\/a> the founder of Wikipedia in Madrid and we spoke about some of these. In general pinging beats crawling in everything but thoroughness. Crawling finds all there is to find on the net, pinging finds what wants to be found. Jimmy described to me a problem that I was not aware of and that is that ajax pages are hard to crawl. I commented on a problem that he was not aware of and that is that Google is the biggest or one of the biggest consumers of electricity in the world and that is among other things because crawling is incredibly energy inefficient compared to pinging. In any case what was extremely interesting is the concept of an open source search engine<\/a>. I really hope that Jimmy and his open sourcers make this one work. One of the worst jobs at Google is probably policing results to make sure they are not hacked as the monetary incentive to hack google results is huge. Wouldn\u00b4t it be great to have a community police force rather than some paid employees? This problem is more manageable than the problem of people who tried to hack Wikipedia. If the Wikipedia community dealt successfully with article hacking, search optimization hacking should also be policed more effectively by a community than by a few paid individuals. Wisdom of the crowds at work in search. Intriguing.\u00a0 In the meantime I mentioned to Jimmy the little search engine that we put together at Fon called Unfolding News<\/a>.\u00a0 This engine combines crawled sources with pinged sources that are all fresh.<\/p>\n

<\/div>
<\/div><\/div>","protected":false},"excerpt":{"rendered":"

I am sure that there\u00b4s tons of stuff written on the web about the pros and cons of pinging (notifications a la technorati) vs crawling (programs that scout the web for links a la google) or listening vs spying. Tonight we had dinner with Jimmy Wales the founder of Wikipedia in Madrid and we spoke […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=\/wp\/v2\/posts\/1013"}],"collection":[{"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1013"}],"version-history":[{"count":0,"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=\/wp\/v2\/posts\/1013\/revisions"}],"wp:attachment":[{"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1013"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1013"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/english.martinvarsavsky.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1013"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}