Blogdigger

» see blogdigger.com as it lived, on the wayback machine

~ the eulogy ~

The first post on Blogdigger's development blog is dated Friday 28 March 2003 at 9:49 in the morning and is refreshingly modest about what it had. Welcome to Blogdigger.com, powered by the RSS Kicker engine. This site is an RSS search engine. Blogdigger is a research project, and is a work in progress. Right now it only indexes about 2,000 weblogs.

Greg Gershman built the thing by himself and kept building. Within two years it had grown a suite: full-text blog search with any results page subscribable as a feed, a media search that pulled MP3 and video enclosures out of RSS, a Groups tool that spliced several feeds into one, and backlink tracking through Link Search. In May 2005 he added Blogdigger Local, which found bloggers within five or a hundred miles of you. FeedDemon and NetNewsWire wired his search into their readers.

The high point came in April 2007, when PC World ran a Search Engine Shoot-Out and Blogdigger finished third in blog search. Gershman's own post about it is almost apologetic: happy to report it faired fairly well, managed to beat out some of the more well-known players. The reviewers liked the clean results page and complained that the index updated slowly, which is what happens when the index is one person's side project.

Nothing about the end is documented. There was no farewell post, no press notice, no status update. The last capture of a working page is an add-a-blog form on 24 May 2016, the last sign of the server answering at all is the next day, and then the archive record goes completely empty for fifteen months on every path the crawler knew about.

~ the dossier ~

~ what rss search was actually for ~

For about four years there was a real question of who would index the blogosphere, and it was not obvious the answer would be Google. Blog posts arrived as feeds, in order, with timestamps, which made them far easier to index in near real time than the rest of the web. A cluster of small companies went after that: Technorati, PubSub, IceRocket, Bloglines, and Gershman working alone in Maryland.

Blogdigger joined the FeedMesh initiative in April 2005 alongside PubSub, an attempt to get the feed indexers sharing update notifications instead of each politely re-crawling the same million blogs. The idea was sound. What actually happened is that Google Blog Search launched, then blogs themselves thinned out into social feeds nobody could crawl, and the whole category lost its subject.

~ it's not going anywhere ~

Odeo acquired Blogdigger on 18 March 2008 for an undisclosed sum, and Gershman joined as VP of Search and Engineering. He posted the news himself with a line that reads differently now: as for Blogdigger, don't worry, it's not going anywhere, Blogdigger will continue to operate as it always has. TechCrunch covered it the same day.

The promise was kept in the most literal possible sense. The site did continue to operate as it always had, for eight more years, with nobody touching it. The development blog went quiet almost immediately after the acquisition post. The footer kept saying Copyright 2006. Gershman went on to become a Presidential Innovation Fellow in 2012, got pulled onto the Healthcare.gov rescue, and co-founded Ad Hoc in 2014, while a JSP application with his name nowhere on it kept serving search results to whoever still asked.

~ eight years with nobody home ~

The archive record of those years is bleak in an oddly specific way. Most of the roughly 4,600 captured hits on the add-a-blog form are automated submissions from feed spammers: counterfeit sneaker pages, casino sites, diet pill blogs. That is what an unattended blog index fills up with. The service was technically alive and was mostly being used by robots to pollute itself.

Then on 25 May 2016 it stopped. The domain reappeared in September 2017 under a new owner as a WordPress coming-soon page, the old JSP paths started returning 410 Gone from somebody else's Apache, and by 2023 blogdigger.com was publishing articles about marmots and dog breeds. Today it resolves nowhere at all: the registration survives but sits on clientHold, which removes it from DNS entirely.

Turkish blogs of the same period are in a worse position, because nothing ever indexed them properly in the first place. A blog search engine dying takes the index with it, and when the blogs go too there is no second copy anywhere. That is most of what happened to the personal web written outside English between 2003 and 2010.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »