Needlebase

» see needlebase.com as it lived, on the wayback machine

~ the obituary ~

ITA Software was a Cambridge, Massachusetts company that wrote the fare search engine most of the airline industry ran on, in Common Lisp, which is the sort of detail that gets a company a reputation among engineers and nowhere else. Somewhere inside it was a side project called Needle, and by January 2010 people were visiting the office specifically to be shown it.

What Needle did was turn scraping into a tagging exercise. You gave it a web page, you clicked the parts of it that were data, and it worked out the pattern and pulled the same fields off every other page on that site. Then it merged what it found with the XML feeds and spreadsheets you already had, using semantic matching to fold variant forms of the same record together. On top of that sat a query language and a set of views. Your edits and merges survived a refresh of the underlying source, which is the part that separated it from a weekend script.

It went public as Needlebase in 2010, free for personal use with a paid commercial tier. The marketing copy asked whether you wanted a faster way to build a vertical search engine without programmers or database administrators, and that was the honest description of the product. ReadWriteWeb's lead blogger called it the WordPress or YouTube of data. The Village Voice used it to run the Pazz and Jop critics poll, and the music editor's blurb on the front page read "nothing short of miraculous".

Google announced its 700 million dollar acquisition of ITA Software on 1 July 2010, and after a Department of Justice review the deal closed on 12 April 2011. The Needlebase blog told users the team would keep operating and innovating as usual, with great new features rolling out that spring. Nine months later, on 20 January 2012, Needlebase appeared on a Google spring-cleaning post. The site went dark on 1 June 2012.

~ the case file ~

~ the note on the door ~

The final page is still in the Wayback Machine, posted by Justin Boyan at ten in the morning on Friday 1 June 2012. It says Needlebase has been retired, and then it does the thing these notices always do and points you at the survivors: Fusion Tables, Google Refine, Public Data Explorer, Freebase, BigQuery, schema.org and the Knowledge Graph.

Read that list in 2026 and most of it is gone. Fusion Tables was shut down in 2019, and Freebase was closed to edits in 2015 before being folded into Wikidata. Google Refine survived only because it was open source and the community carried it on as OpenRefine, which is to say that the healthiest item on Needlebase's list of successors is the one Google could not switch off.

~ what it cost to do this by hand ~

In 2012 getting structured data off a web page meant writing the scraper yourself. You found the pattern in the markup, wrote the selectors, handled the pagination, and then the site redesigned and you did it again. I have written that code. Everyone who has done this job for long enough has written that code, usually more than once, usually for data that stopped mattering before the script stopped breaking.

Needlebase's claim was that this should be a user interface problem rather than a programming problem, and the demos it built to prove it were music polls and festival listings rather than anything enterprise. A decade and a half later the pitch has come back under different names, and the products making it now are trained models rather than tagging interfaces, but the problem has not changed at all.

~ why google bought it ~

Google did not buy ITA Software for Needlebase. It bought a fare search engine to put flight results directly into its own search pages, which is what Google Flights is, and Needlebase was an asset on the balance sheet of the company being purchased.

The retirement notice said the technology was being evaluated for integration into Google's other data-related initiatives, which is the standard formula and occasionally even true. Whatever happened to the code, the product with a login page and paying customers had fourteen months of life after the deal closed.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »