Google Flu Trends

» see google.org as it lived, on the wayback machine

~ the eulogy ~

a model that beat the cdc by two weeks, published in nature, and then spent five years getting slowly wronger while everyone watched. i never loaded the page. the country list ran east as far as russia and ukraine and stopped there, so there was nothing here to look at. what i did read, later and more than once, was the postmortem, because it is the sort of failure you end up thinking about if your working life involves models that were correct on the data you had.

the idea itself was sound and is still sound. somebody with flu searches before they go to a clinic, and the clinic's report reaches the centers for disease control a week or two after that. if you can find the queries that track the reports, you get the same signal earlier and for free. google.org put it up on 11 november 2008 for the united states and grew it out to twenty-nine countries, with a parallel dengue product covering another eleven.

in february 2009 nature carried the method under jeremy ginsberg's name, along with a co-author from the cdc's own influenza division. the model was a linear fit against historical clinic data, using the forty-five search terms that correlated best out of fifty million candidates. the paper reported the fit at around ninety-seven percent. for a while it was the most quoted example of a company doing public health work by accident.

the failures started almost immediately. it did not see the 2009 h1n1 wave coming in the spring, which was the one event the whole thing existed to catch. between 2011 and 2013 it ran consistently high, and in february 2013 nature reported that flu trends was estimating more than twice the share of doctor visits the cdc actually recorded.

~ the dossier ~

~ the drift ~

the diagnosis published in science in march 2014 by lazer, kennedy, king and vespignani is worth reading in full, and its argument is not that big data is bad. it is that the input was never a stable measurement. the forty-five terms were chosen because they correlated, not because anybody could say why, and some of them were picking up searches about winter illnesses that are not flu at all. the authors also noted that google would not publish the actual query list, which made independent correction impossible.

the deeper problem is that google kept changing google. autocomplete suggestions, related searches, the whole machinery of search itself moved under the model, and every one of those changes altered the volume of the exact terms the model depended on. a search engine improving its own product was quietly corrupting a dataset that assumed searching stayed constant. the team retrained in 2009, again in 2013 and again in 2014, which is the shape of a system being patched rather than fixed.

~ the last estimate ~

the last current estimate went up on 9 august 2015. eleven days later the flu trends team posted a note saying they would stop running their own site and hand the raw signal to people who do infectious disease research for a living, naming columbia's mailman school and the cdc's influenza division among the recipients. the note said it was still early days for nowcasting, which is a fair thing to say after seven years and a lot less fun than the nature paper.

the historic estimates were left on the page for download, so the whole run from 2003 onward is still sitting there, along with the three different models that produced different numbers for the same weeks. that archive is more useful than the live product ever was. it is the closest thing anybody has to a public record of a prediction system failing in slow motion with its own homework attached.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »