Gigablast

» see gigablast.com as it lived, on the wayback machine

~ the last words ~

There is a number for how many organizations actually crawl the open web and hold their own index of it, and the number is small enough to list on one hand. Everything else with a search box, including most of the privacy-branded engines people switch to, is buying results from Google or Microsoft and putting a different face on them. Gigablast was on the short list. When it went dark in April 2023 the list got shorter by one, and it happened with no press release and no goodbye page.

Matt Wells had worked at Infoseek, took a computer science degree out of New Mexico Tech, and went back to Albuquerque to write a search engine in C and C++ by himself. He founded it in 2000 and opened the public beta on 21 July 2002. The stated ambition on the site was an index of two hundred billion pages, which was preposterous and also the correct thing to aim at, since an index that stops early is an index that misses the thing you are looking for.

It never got near two hundred billion. By 2015 the claim was better than twelve billion pages, which is a small fraction of Google and an enormous number for one person with a rack of machines. The engine did things the big ones had stopped bothering with, including strict boolean operators that actually worked and a feature called Giga Bits that pulled related concepts out of the result set. In 2010 the site was advertising that it ran on ninety percent wind power, which tells you the operation was small enough for one person to make that decision on a Tuesday.

The reason I never used it is the reason most non-English speakers never used any of these. What an independent crawler gives you is its own index, and its own index in this case was the English-language web assembled by a crawler running out of New Mexico. Type a query in Turkish and there is nothing behind it. That question was settled here early, first by the Turkish portals that indexed Turkish pages, then by Google being adequate at everything, and by the time I had broadband there was no reason to open a second search box at all.

~ the record ~

~ one person, one index ~

The costs of what Wells was doing are worth stating plainly. Crawling means fetching billions of pages politely and going back to check whether they changed, then building an inverted index over all of it and serving queries fast enough that a stranger does not close the tab. Google does this with data centers. Gigablast did it out of Albuquerque on hardware Wells owned, which is why the numbers were what they were and why the results were sometimes stale.

The business underneath it was licensing. Gigablast supplied results to other people's front ends over the years, Ixquick and Clusty among them, and it powered the Internet Archive's site search. In November 2019 it became the index behind Private.sh, a search engine run with Private Internet Access that encrypted the query on your machine and passed it through a proxy so that neither side could assemble a profile. That partnership is the clearest statement of what an independent index is actually for. You cannot promise anybody privacy from Google while asking Google for the answers.

In July 2013 Wells put the whole engine on the internet under Apache 2.0. The crawler and everything built on top of it, free for anyone to run. That was a strange thing to do with the only asset he had, and it is the reason the software still exists even though the service does not.

~ what is left ~

Nothing was announced. The site stopped responding in the first days of April 2023, and the last snapshot in the Internet Archive that still returns a working page is dated 31 March. Wells said nothing publicly, no acquisition was reported, and the domain simply stopped resolving. Mojeek, one of the few remaining operators of an independent index, put up a short farewell on 3 May 2023 that was mostly about what the closure meant for everyone else.

That is the part with weight. A search engine dying is normally a story about a company running out of money, and this one has no company in it. Twenty years of one person keeping robots on the open web, licensing the output to whoever wanted an alternative, and then one week the machines stopped and the only evidence is a domain that no longer answers. The code is still up. Anybody can read exactly how it worked, which puts Gigablast in the small category of dead software that left instructions.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »