W3 Catalog

» see cui.unige.ch as it lived, on the wayback machine

~ the obituary ~

the first search engine on the web was switched off two years before i got a modem, which means i have never once used a web that did not have a search box somewhere in it. that is worth stating because it is a very short window. the web was searchable within four months of most people being able to see it at all.

the chain of events is small and specific. ncsa released mosaic in the spring of 1993 and the web stopped being a text terminal thing. cern was up the road from the university of geneva, so geneva invited tim berners-lee over to give a seminar with a live demo, and set up their own server for it, announced publicly on 25 june 1993.

then oscar nierstrasz ran into the problem that the cern http server was difficult to configure for pages whose output was generated rather than stored. on 10 august 1993 he switched the machine to tony sanders' plexus server, which was written in perl and therefore easy to extend. he wrote a small perl package called parscan that would return every html paragraph in a file matching a query string. that was the whole search technology. it was released on 2 september 1993.

on the same day he pointed it at something. a number of people had been hand-maintaining good lists of web resources, cern's www virtual library among them, and the only thing wrong with those lists was that you could not search them. he wrote one script to mirror the lists and another to cut them into individual chunks, then glued every chunk into a single searchable file with parscan in front of it. it was briefly called jughead and then renamed w3catalog, and the announcement went out that afternoon.

~ the case file ~

~ how it actually worked ~

there was no crawler and there was never going to be one. resources appeared in w3catalog because a human being had put them in one of the lists it consulted, and there was no way for anybody to register a site directly. nierstrasz says this was deliberate rather than an oversight.

almost nothing changed over three years. lists were added to the set. the chunking script kept getting rewritten because the lists kept changing format, which was the permanent weak point of the whole approach. the search back end went its own way and outlived the catalog by a long distance: in october 1993 nierstrasz packaged it to run under htbin, then the cgi standard arrived and made that obsolete within weeks, and on 12 may 1994 parscan came back as a generic cgi script called htgrep, which ended up sitting behind a great many other search boxes around the web.

~ why he stopped ~

the history page he wrote lists four reasons and none of them is sentimental. the first is that the catalog rebuilt itself automatically and somebody still had to keep track of which lists it was consulting and whether the urls in them were still good, on top of answering the mail. his exact words: nobody has time or interest for this.

the second was fragility, because a small change in the format of somebody else's list would break the chunker and fill the catalog with records that ran for pages or were not valid html. the third was arithmetic. every single query started a new process, loaded the perl compiler, compiled htgrep, and then read and scanned the entire database, which was several megabytes by then. at busy times several requests were arriving every second. in his phrasing, this quickly brings the cui server to its knees.

the fourth reason he simply concedes. altavista and hotbot were periodically scanning the whole web, and their implementations were not cgi scripts but heavily optimized machinery. much better and much more comprehensive search engines were now available, he wrote, and left it at that.

~ the note he left ~

the last section of the history page is headed is there a life after death, and the answer begins: well, in this case, i don't think so. w3catalog has had its run. he notes it got referenced in most of the early books about the web, which is true and is also the sort of thing you say about a career rather than a service.

what he does not do is pretend the idea was wrong. there is still a need, he writes, for a search engine built on high-quality human-managed lists, and the reason nobody has built one is that a common format would be needed and list maintainers will not adopt one if it makes more work for them. anybody who has kept a list will sympathize with that, he adds. the page is signed and dated 8 november 1996, which is also the day the catalog stopped answering.

the domain w3catalog.com was picked up by other people in february 2010 and run as a directory where every entry was reviewed by hand, which is a genuine tribute even if it had no connection to geneva. it went quiet again by 2023. the actual inheritance is elsewhere and it is not the search box. it is dmoz and every curated list since, all of them running on the same bet nierstrasz made in 1993, that a person who knows a subject sorts it better than a machine that has merely read it.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »