SciGraph

» see scigraph.springernature.com as it lived, on the wayback machine

~ the obituary ~

SciGraph was a publisher giving away its own metadata. Springer Nature launched it in March 2017 as a linked open data platform: every paper, author, funder, grant and conference in its catalog, expressed as RDF statements that a machine could follow from one to the next. The first release ran to about a billion of those statements and later versions roughly doubled it, covering something like thirteen million publications, the oldest of them from 1815.

The point was that scholarly metadata had always been locked inside whoever published the paper. You could read an article, but asking a question across the whole corpus meant paying someone for a database, and the answer you got back was shaped by their schema. SciGraph published the graph itself under an open license, with a SPARQL endpoint for querying it live and bulk downloads for anyone who wanted the lot.

Michele Pasin was the data architect and technical lead, working with a group inside Springer Nature that had to argue the open data case internally before any of it could ship. The stack underneath was the usual research-infrastructure mixture of document stores and search indexes, with Elasticsearch doing the heavy lifting on the explorer interface. At its height the platform was serving twenty to thirty million hits a month.

Semantic web technology was being described as the obvious future of everything for most of my career, and in practice I have watched almost every system I worked on solve the same problems with a database and a well-chosen schema instead. SciGraph is the counterexample people in that field point at, because somebody with real data and real money built it properly and shipped it. It ran for six years and then the publisher turned it off.

~ the case file ~

~ a billion triples ~

The scale was the argument. A billion statements about thirteen million papers is enough to ask questions nobody had budgeted for: which funders show up together, how a subject area moved between institutions over thirty years, what a journal's citation neighborhood actually looks like. Researchers in bibliometrics used it precisely because it was free and complete rather than sampled, and those are the two properties commercial citation databases never offer at once.

There is a tell in the public repository. The GitHub release folders are labeled by quarter, and they stop at 2018Q2. The bulk data dumps had quietly ended almost five years before anyone announced a shutdown, which is usually what the end looks like from the inside: not a decision, just a thing that stops being someone's job.

~ where the data went ~

The last release did not vanish, because the team put it on Figshare before the lights went out and described it as remaining available indefinitely. The GitHub repository is still up with its documentation. Pasin wrote a retrospective three days after the retirement, calling the project a serious attempt to bring new technology into a working commercial setting.

What is gone is the live part. The SPARQL endpoint answered arbitrary questions against current data, and a frozen dump on a file-hosting service does not do that. Anyone who built a tool on top of the endpoint had their tool break on 31 January 2023, and the replacement is to download several gigabytes and host the query engine yourself. Springer Nature never published a real explanation for the retirement.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »