BotSeer

» see botseer.ist.psu.edu as it lived, on the wayback machine

~ the goodbye ~

In 2007 a group at Penn State built a search engine whose entire index was robots.txt files. The index was not the pages behind those files but the files themselves, 2.2 million of them, harvested from 13.2 million websites and searchable by what they said and who they said it to. BotSeer ran out of the College of Information Sciences and Technology and was open to anyone who found the address.

You could search the corpus, and you could run the bias analysis that was the actual point of the exercise. A prototype also let you search six thousand documentation and source files drawn from eighteen open source crawler projects, which made it a reference for people writing crawlers as well as for people trying to keep them out.

The bias analysis is what got it into the newspapers. In November 2007 the Associated Press and Network World both reported the result: site owners were writing more permissive rules for Googlebot than for the crawlers behind rival search engines. Network World ran it under a headline saying Google was favored by web administrators. The file that every crawler was supposed to read had been getting written with one crawler in mind.

Nobody announced a closure. The capture record is the only obituary here. The site answers with a normal page from July 2007 through 2 July 2010, and by July 2011 the same address is returning a stub just over half that size. It is serving errors by 2013. Today the Penn State host does not respond at all and botseer.org hands back an empty body of 114 bytes. The working life is given as roughly 2007 to 2010 and the captures agree with that.

~ the chronicle ~

~ the objection ~

Not everyone welcomed it. The argument against BotSeer was that robots.txt works because nobody looks too hard, and that pulling every one of those files into a searchable corpus cut against the convention it was studying. A line asking politely that you stay out of one directory becomes, in aggregate, a map of what site owners would rather nobody saw.

The counter-argument is that the files were public by definition. Every one of them sits at a fixed, guessable path and is served to anyone who asks for it. BotSeer read what was already published and set the readings side by side.

~ the honeypot ~

The project also ran a trap. It set up a honeypot to watch how crawlers behaved in practice as opposed to how their documentation claimed they behaved, and specifically whether they honored the exclusions they were served. That part of the work aged into ordinary practice. Checking whether a bot actually obeys the file is routine infrastructure work in 2026, and the current argument about AI crawlers ignoring it is the same argument with a much larger bandwidth bill attached.

The last working page is dated 2 July 2010. The files BotSeer indexed are mostly still sitting at the same paths, saying similar things to an entirely different set of robots.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »