World Wide Web Worm

» see cs.colorado.edu as it lived, on the wayback machine

~ the obit ~

In march 1994 I was a child in Ankara and four years away from my first modem. The web I eventually arrived on already had search boxes in it and I assumed they had always been there. They had not. Somebody had to be first to point a program at the web and let it walk, and one of the people who did that was a numerical analysis professor in Colorado who was not trying to start anything.

Oliver McBryan wrote the World Wide Web Worm at the University of Colorado at Boulder in september 1993 and released it in march 1994. It came out of an earlier project of his, a directory of web content that people could extend by hand, which was how the whole web was catalogued at the time. The Worm was the version that did not need people. It followed links and put what it found into a database anyone could query.

The numbers look small now and were enormous then. About 110,000 documents indexed, sitting in a database of roughly 300,000 multimedia objects. In 1994 that was a substantial fraction of everything that existed. McBryan presented it in a paper called "GENVL and WWWW: Tools for Taming the Web" at the first International World Wide Web Conference at CERN in Geneva, over three days that may, and the Worm took the Best Navigational Aid award at Best of the Web '94.

One design decision from that paper outlived everything else about it. The Worm indexed the text inside links, not just the text of the page a link pointed at. It is the first search engine known to have done this. Four years later two students at Stanford built a ranking system on top of the same observation, that a link is a statement made by one person about another person's page, and the entire economics of the modern web came out of it.

~ the rap sheet ~

~ before anybody crawled ~

It is difficult now to picture how the web was navigated in 1993. You read a list. Somebody at a university kept a page of interesting new sites and updated it by hand, and you visited that page because there was no other route in. Yahoo started this way and stayed that way for years, with human editors filing sites into categories.

The argument for hand-built directories was quality, and for a while it was a good argument. The argument against was arithmetic. A person can file a hundred sites a day and the web was already growing faster than that in 1994. Crawling was not a better idea than editing, it was the only idea that scaled, and a couple of academic projects worked that out before the industry did.

~ regular expressions in the search box ~

The Worm's query language was Perl regular expressions. You could search the words inside links, or the titles and addresses of documents themselves, and you could express what you wanted with the full syntax of a pattern language written for programmers.

This tells you exactly who the users were. Everybody on the web in 1994 was technical, so a tool built for technical people was not a limitation, it was just correct. It also explains why nothing like it survived contact with the general public three years later. The winning interface turned out to be one empty box that accepts anything typed into it badly, and every product that asked the user to be precise lost to the product that asked the user for nothing.

~ what goto did with it ~

Around 1997 the technology went to GoTo.com, the Bill Gross company built inside Idealab in Pasadena. GoTo's idea was that search results should be sold: advertisers bid on a term and the highest bidder appeared first, with the price shown next to the listing. It was treated as vulgar at the time. It also worked, and it is the business model that pays for search to this day.

GoTo renamed itself Overture in 2001 and Yahoo bought it in 2003 for over a billion and a half dollars. So the Worm's code went into the machine that turned search into an auction house, while the man who wrote it stayed at the university. McBryan said later that it had been an educational project and that commercializing it never occurred to him, partly because there was no department around him that dealt with this sort of thing. The original page at cs.colorado.edu stopped answering some time after that, and nobody wrote down when.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »