Findx

» see findx.com as it lived, on the wayback machine

~ the obit ~

Findx was a search engine from Denmark that did the expensive thing. Most privacy-branded search products buy their results from Microsoft or Google and put a privacy policy in front of them. Privacore, the small company behind Findx, ran its own crawler and built its own index instead, on servers in Denmark with no query logging. It put up a teaser page in August 2015 that spelled out RESPECT and PRIVACY in binary digits, then opened the engine for beta access in 2016.

The code underneath was a heavily modified fork of Gigablast, the open source engine Matt Wells had been running since the turn of the century. Privacore stripped out about half the original source and moved the spider database onto sqlite3. The fork stayed public on GitHub for the whole run, so what they changed is still readable today.

Findx indexed for Europe and the English-speaking web, so a Turkish query was never the sort of thing it was built to answer well. I found it the slow way, years after it closed, by following a dead redirect to a GitHub page that turned out to be dead too.

On 8 November 2018 Privacore changed the description on its search engine repository to read "No longer maintained. Please read our shutdown message," and published that message on GitHub Pages. The README got a new first line above everything else: "Warning: Do not use this code."

~ the rap sheet ~

~ the index problem ~

What Findx set out to do was harder in 2016 than it had been in 2000, and not for technical reasons. Large sites had spent fifteen years tuning their robots files around exactly two crawlers. Yelp and LinkedIn would not let Findx in at all. Then a bug in Privacore's own crawler made its machines look like an attack, and several of the largest infrastructure providers blocked the address ranges outright, which cost the index whatever those networks were fronting.

Most of what did get collected was spam. Running an independent crawler in the late 2010s means paying full price for a web that has been optimized against you by everyone with something to sell. In September 2018 Privacore forked searx, the open source metasearch frontend. That is the move you make once you have decided to stop crawling and start borrowing results. Eight weeks later the company was gone.

~ the page it pointed at ~

The farewell is one of the better ones and almost nobody has read it. The headline is "Goodbye, Findx is shutting down," the structured data carries a name field reading "Findx - Game over," and the opening line is "It is over, before it really started." Brian Rasmusson signed it as founder and left his findx.com address at the bottom of a page hosted on privacore.github.io.

Here is why it is hard to find. By late January 2019 findx.com had been reduced to a four-line HTML file whose only content was a meta refresh pointing at privacore.github.io, and GitHub had stopped serving that address on 10 November 2018, two days after the farewell went up. For years afterward anyone typing the dead search engine's domain into a browser landed on GitHub's "There isn't a GitHub Pages site here" screen. The goodbye outlived the search engine by about forty-eight hours, and the redirect to the goodbye outlived both by a decade.

~ what the farewell blamed ~

Two passages are worth keeping. The first reads like an honest product post-mortem: "We've learned that people say one thing about privacy, and do something else when it comes to privacy vs. convenience." The second is less generous. "Another thing that surprised us a bit was the resistance shown by tin-foil-hat privacy activists... could any tool ever be good enough for them?"

A privacy company closing its doors while complaining about privacy activists is a particular kind of ending, and the complaint is not entirely unfair. Findx had been a verified publisher in Brave and a built-in search option in Waterfox, so the people most likely to install it were also the people most likely to audit it and declare it insufficient.

The farewell also thanks Greg Lindahl of Blekko for advice on crawling and indexing, and wishes him better luck than Privacore had. Blekko had been gone since March 2015 by the time that line was written.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »