Scrub The Web

» see scrubtheweb.com as it lived, on the wayback machine

~ the eulogy ~

In May 2017 the front page of scrubtheweb.com lost its search box. What went in its place was a notice saying that after more than 21 years online the domain was for sale, and that the engine, the software behind it and the customer list were not part of the offer. By early June the same banner carried a date. The site would officially shut down on 18 June 2017.

It had started in 1996 as a directory, added a crawler, and then spent two decades doing the job that made the early web navigable. You typed your address into a form, a robot came around at some point and read your pages, and weeks later you were in an index. Scrub The Web ranked what it found on the words in the document rather than on how many other sites linked to it, which was ordinary practice in 1997 and effectively extinct by 2005.

The engine came out of a small Arizona software house called Absolute Business Systems, and by the account of the one site that bothered to write up its closing, most of it was the work of a single programmer. Long before the end the search results were beside the point. Traffic came from webmasters using the free meta tag tools and the how-to pages, and the money came from a paid membership called Scrubby Prime Plus. The crawler had become the expensive organ in a body that was really an SEO shop.

The message to registered users said plainly that running the engine had gotten too expensive, and that the tools would stay available until existing contracts ran out. No lawsuit, no acquisition, no last-minute buyer for the index itself.

~ the dossier ~

~ the submit url ritual ~

For about ten years, getting a website found meant filling in forms. Every engine had an Add URL page, you worked through them one at a time, and then you waited to see whether the crawler had come. Scrub The Web ran one of the handful of submission forms people still bookmarked well into the 2000s, long after the practice had stopped deciding anything.

In Turkey the same ritual pointed at different addresses. Around 2000 the way to get a page seen here was to talk your way into the category tree at Arabul or Netbul, because a search that could be told to look only at sites from this country beat a crawler in Arizona that had never heard of any of them. Google ended the argument for everybody. Links did the submitting from then on, and the Add URL page became furniture that a few engines kept out of habit.

~ ranking without popularity ~

The claim Scrub The Web kept on its front page for twenty years was that it did not weigh how popular a site was. A page that matched your words could come first whether ten people or ten million had ever visited it. Read one way that is the fairest possible search engine. Read another way it is a system anyone can climb by writing the right words into a meta tag, which is exactly what the site's own tools taught people to do.

Every engine that survived past 2002 went the other way and started counting links, because counting links is harder to fake than counting keywords. Scrub The Web kept its method and kept its audience of webmasters, and the audience for its actual search results thinned out to almost nothing.

~ the address afterward ~

By July 2017 the domain had changed hands and was serving a paid web directory: submit your site, buy a listing, get a link that passes ranking value. That is still what sits there in 2026, advertising a dofollow link and the line "active since 1996," which is true of the domain name and of nothing else on the page.

The crawler and the tools were never for sale, so they went wherever software goes when a small company stops paying for servers. What got resold was the one asset with a market: an address old enough that search engines still trust it.

credit to a reader who wrote in.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »