WebCrawler

» see webcrawler.com as it lived, on the wayback machine

~ the goodbye ~

by the time i got online in 1998, webcrawler had already had its moment and lost it. the name still floated around, but the search i actually used was altavista and then very quickly google. webcrawler belonged to an earlier slice of the web that i can read about but was not there for. it launched on 21 april 1994, built by Brian Pinkerton, a computer science student at the university of washington, in the hours around his actual coursework.

the thing that made webcrawler matter is easy to miss now, because every search engine does it. it read the full text of the pages it crawled, not just their titles or some hand-written description. before that, finding a page meant hoping someone had listed it under the right heading. webcrawler let you search for words that appeared anywhere on a page, which sounds obvious and was not. on 14 november 1994 it answered its millionth query, and the web it indexed was still small enough that one student could keep up with it.

then the money arrived. america online bought webcrawler on 1 june 1995 and made it the default search for a large chunk of people meeting the web for the first time through the aol client. two years later aol sold it to Excite for around twelve million dollars. Excite ran it as a separate engine for a while, then quietly wired it to its own index, and the independent webcrawler was effectively over by 2001. that same year Excite@Home went bankrupt and the name got passed to InfoSpace like a piece of furniture in an estate sale.

~ the chronicle ~

~ what it actually did ~

webcrawler crawled outward from a seed list of pages, followed every link it found, and stored the words of each page in an index you could query. the interface was a box and a list of results, which is still the interface. what changed underneath was scale and ranking, not the shape of the thing. Brian Pinkerton wrote the first version as a desktop application before it was ever a website, which tells you how small the idea started.

~ what it became ~

webcrawler.com still resolves in 2026. a company called System1 owns it now, the same outfit that ended up with dogpile and metacrawler, and it serves ads dressed up as a search page. none of that is webcrawler. the engine that read the whole web first stopped existing the moment it started borrowing someone else's index. it sits next to [[lycos]] and [[infoseek]] in the short list of search engines that were first at something and got remembered for nothing in particular.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »