ScraperWiki

» see scraperwiki.com as it lived, on the wayback machine

~ the last words ~

In December 2009 the venture arm of a British television channel invested in a startup that wanted to make web scraping into a shared activity. 4iP was Channel 4's public-service innovation fund, ScraperWiki was a Liverpool company founded that year by Julian Todd and Aidan McGuire, and the pitch was in the name. Scraper, for the programs that pull data out of a website that never meant to publish it as data. Wiki, for the idea that anyone who could code should be able to open somebody else's scraper and fix it.

The timing was good. Britain was two years into a data-journalism boom that had the Guardian's datablog at the front of it, councils were being pushed to publish spending files, and newsrooms had discovered they employed nobody who could open a 40,000-row CSV. ScraperWiki ran hacks-and-hackers days where reporters and programmers were locked in a room together for eight hours, and the Irish Times covered one of them in 2010 under a headline about getting solid stories out of difficult data. Tim Berners-Lee told a Guardian conference that November that analyzing data was the future for journalists. The platform was standing in exactly the right place.

What it actually sold was a hosted environment. Your scraper lived on their servers and ran on a schedule without a machine of yours being awake. The output sat in a database at a URL somebody else could read. Later it added an in-browser Python and R workbench aimed at economists and statisticians who were new to code, which is how the Office for National Statistics and the Department for Communities and Local Government both ended up as case studies.

Then it got a second funding round from Enterprise Ventures, and then it got quieter.

~ the record ~

~ the two new names ~

Around 2016 the company split its identity in half. The product became QuickCode, on the stated grounds that it was no longer a wiki and no longer only for scraping. The company became The Sensible Code Company, which was a fair description of a business now selling PDF table extraction and data tooling rather than a community scraping platform.

Load scraperwiki.com and you still get the announcement page from that year, complete with the line "ScraperWiki has two new names!" and two buttons sending you off to the product and to the company. It is a ten-year-old redirect notice that has outlived both of the things it was redirecting you to.

~ what Sensible Code sells now ~

QuickCode's own site was still selling the product in January 2019. By August that year the same page carried a meta description reading "QuickCode has now been decommissioned", pointing visitors at PDFTables and at something called TableBuilder. That notice sat there for three more years before quickcode.io was reduced to a meta refresh, which is what it is today: eleven lines of HTML that bounce you to sensiblecode.io.

Follow that bounce and you land on cantabular.com, because The Sensible Code Company renamed itself Cantabular Ltd. Cantabular sells statistical disclosure control, which is the software that lets a census office publish tables without letting anyone identify a household inside them. Aidan McGuire is the founder and Julian Todd sits on the board as a non-executive director. The customers are the ONS and its Northern Irish counterpart. The company that built a tool for prying public data loose now builds the tool that decides how much of it is safe to release.

~ the patience part ~

ScraperWiki's bet was that a scraper is a shared artifact, worth hosting properly and worth handing to a stranger. That is not how most of us treat them. A scraper is forty lines you write at eleven at night against a page that will change its markup in March, and it lives in a folder on your own machine with a name like temp2.py, and you rewrite it from scratch the next time rather than read what you did before. The platform was asking scrapers to be code. They are mostly disposable stationery.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »