Scitopia

» see scitopia.org as it lived, on the wayback machine

~ the obit ~

Twenty-one scientific societies built a search engine together. It ran for four years and seven months, and the software was the easy half.

Scitopia went into beta in June 2007 and out of beta that October. The idea was federated search: instead of copying everybody's content into one index, the site sent your query out to each partner's own library in real time and merged what came back. Behind it sat the IEEE, the American Society of Mechanical Engineers and nineteen other learned societies, plus roughly fifty million patent records drawn from three national patent offices. The engineering came from Deep Web Technologies, whose Explorit platform did the actual federating; the societies supplied the libraries and the permission.

Permission was the whole product. Every one of those societies sold subscriptions to the content being searched, and each had a reason to keep its own search box as the front door to its own journals. Getting twenty-one of them to point at a shared box, free and with no paywall on the result list, was a negotiation that had to be won separately twenty-one times. They won it in 2007, which was also the year Google Scholar stopped looking like an experiment.

~ the rap sheet ~

~ what it was actually searching ~

More than three million documents, the site said, spanning hundreds of years of science. That number is small next to a crawler and enormous next to what you could reach in 2007 without an institutional login, because every document in it was peer reviewed and every one came from the body that published it. There was no ranking argument to have and no spam to filter. The index was the membership list.

Federated search also meant the thing was only ever as fast as its slowest partner, and a query had to wait for twenty-one servers of wildly different vintage to answer. People who used it remember it as deliberate rather than quick. Nothing in the result list was ever stored on Scitopia's own disk.

~ the thursday evening timestamp ~

The portal's last working capture is from 2 January 2012, a full page at about ten kilobytes. The next one, from 11 January, is half the size and carries a different page entirely: "The Scitopia partner societies regret to announce that the Scitopia free federated search portal has been discontinued." Two sentences of thanks follow, then the full list of twenty-one societies with their own addresses underneath, each one described in the present tense.

Replaying that capture with the headers intact returns last-modified: Thu, 05 Jan 2012 18:27:26 GMT. Somebody saved that file on a Thursday evening, and the date matches the one the record has always carried without a source attached. The server kept the page up for another month and started answering 503 in February. By October 2012 it had stopped answering at all, and today scitopia.org does not resolve.

~ why nobody noticed ~

This was built for someone like me. In 2007 I was still inside the university years in Ankara, reading papers because the degree demanded it, and a free box that searched the IEEE and the ASME at once belonged on my bookmarks bar. I never heard of it. Geography had nothing to do with that. By 2007 the question "where do I search for papers" already had a default answer, and anything other than the default had to be found on purpose.

No reason was ever given for the closure. The notice says discontinued and thanks you, and that is the complete public record of why twenty-one societies stopped cooperating. Each of them was paying a share of a portal whose traffic had stopped justifying the share, and all twenty-one still had a search box of their own.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »