ALIWEB

» see aliweb.com as it lived, on the wayback machine

~ the obit ~

On 30 November 1993 a software engineer at Nexor in Nottingham posted a note about something he had built called ALIWEB. The name stood for Archie-Like Indexing for the Web, which tells you exactly what it was copying. Archie had been indexing anonymous FTP servers for a couple of years and Martijn Koster wanted the same service for this new hypertext layer that was starting to have more than a few hundred sites on it.

There were perhaps six hundred web servers in the world that autumn. Nobody had solved the problem of finding anything on them, and the obvious solution, sending a program out to read every page, looked reckless to the people running those servers. Bandwidth was expensive and a badly written robot could knock a machine over. Koster understood this better than most, since he went on to write the robots exclusion standard that every crawler still honors.

So ALIWEB did not crawl. It asked. A site owner wrote a small index file describing their own pages, in the IAFA format borrowed from the FTP world, and registered where that file lived. ALIWEB fetched the registered files on a schedule and merged them into one database you could search. It was cheap to run, and it put the description of a site in the hands of the person who actually knew what was on it.

I came online in 1998 on a modem, five years too late to have met any of this. By then finding things meant Yahoo's directory and then AltaVista, and the idea that a search engine might wait politely for you to describe your own site would have sounded like a joke. ALIWEB sits in the part of the record I can read about and cannot remember, which is most of the web before I had a phone line to it.

~ the rap sheet ~

~ the submission problem ~

The thing that made ALIWEB elegant is the thing that killed it. Asking site owners to write and host a structured index file assumed a web where the people running servers were the same people who cared about being found. That held for about a year. Once the web opened out past universities and research labs, most people putting up pages had no idea what an IAFA template was and no reason to learn one.

Koster presented the design at the first international WWW conference at CERN in May 1994, in a paper titled "Aliweb, Archie-Like Indexing in the Web". The audience was the right one and the argument was sound. It did not matter. Submissions never reached the volume that would have made the index worth searching, and by 1995 the crawlers had won on the only measure users cared about, which was whether the thing you wanted was in there.

~ the bug nobody found in time ~

There is a smaller failure sitting underneath the big one. The database file grew past what the search code could scan in a single pass, so results were weighted only across the portion it managed to read before it stopped, rather than across everything ALIWEB actually held. A later operator fixed it, well after the point where a fix would have changed anything.

It is a familiar shape for a first attempt at a category. The idea was right enough that everything after it borrowed something, and the implementation was fragile enough that none of the borrowers ever had to compete with it.

~ the long tail ~

Here is the part worth keeping. ALIWEB stopped mattering in the mid 1990s and did not stop existing. The aliweb.com page kept loading with its banner ad slot and a header reading "The Web's Oldest Search Engine, Est. 1993", along with a link to a September 11th memorial page that had been sitting there since 2001. The Internet Archive holds it in that exact state through the winter of 2025.

The last capture with the real page on it is 16 March 2025. By 22 June the address returned a redirect, and it now lands on a domain broker's sale listing. Somebody stopped paying, or a renewal notice went to an address nobody reads anymore. AltaVista and Infoseek were switched off by corporations that filed paperwork about it. The first one just quietly went unrenewed, thirty-one years after it was announced.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »