World Wide Web Wanderer

~ the obit ~

The Wanderer stopped counting about three years before I got online. When I first heard a modem negotiate in 1998, the web was already a place you needed a search engine to survive, and the question of how many sites existed had stopped being a question a person could answer. For a window of two years it had been answerable, and one graduate student at MIT was the one answering it.

Matthew Gray deployed it on 30 June 1993. It was a Perl program that followed links outward from a starting point and wrote down what it found, and its purpose was measurement rather than search. In June 1993 it came back with 130 web sites in existence. Gray ran it monthly after that, and the numbers it produced are the source of nearly every chart of early web growth you have ever seen, ending around 23,500 sites in the middle of 1995.

Along the way it had to keep a record of the pages it had visited, and that record got a name, Wandex. Because it was a list of pages you could query, it has a decent claim to being the first web search engine, made by accident, by somebody who was trying to do statistics. Gray did not push it in that direction and never turned it into a business, which is why the name in the history books is usually somebody else's.

~ the rap sheet ~

~ what it measured ~

The number worth sitting with is 130. In the summer of 1993 the entire World Wide Web was 130 servers, which is small enough that a determined person could have visited all of them in a week. Two years later it was over twenty thousand and doubling on a schedule, and the honest conclusion of Gray's own project was that this method had an expiry date.

Automated crawling was also new enough in 1993 to be controversial. Robots hitting a server were an unbudgeted load on a machine that somebody was paying for, and the argument got heated enough that the robots exclusion standard, the robots.txt file every crawler still reads, was proposed in 1994. The etiquette layer of the modern web was written while the first crawlers were still running.

~ the funny part ~

Gray built the first thing that walked the whole web, produced the first index of it as a byproduct, and then went back to what he was actually doing. The commercial search engines arrived a year or two later with venture money behind them and took the category. He later went to work at Google, which is a tidy ending for the person who wrote the crawler nobody made money from.

What the Wanderer leaves behind is a habit rather than a product. Every uptime monitor and every crawl-based dataset since is doing the same thing it did, which is to go look instead of asking somebody for the number. In 1993 that was a novel enough idea to need writing from scratch in Perl.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »