Mozilla Test Pilot

» see testpilot.firefox.com as it lived, on the wayback machine

~ the goodbye ~

test pilot's peak years overlapped with my firefox years, but i never signed up. the program was the way mozilla shipped half-finished features to a self-selecting audience without committing the rest of the user base to them. you went to testpilot.firefox.com, you saw a grid of small experiments, you installed one, you got a feedback prompt every couple weeks. that ask, opt in to a thing the engineers might pull later, was not how i wanted to use a browser. so i read about the experiments after the fact when they got blogged about.

mozilla launched the second-generation program in may 2016 after a first version had run as a more lightweight add-on a few years earlier. the second-generation pitch was discrete. you wanted a screenshot tool, you wanted a clean way to log into facebook without bleeding cookies into the rest of the browser, you wanted a sidebar full of vertical tabs, you wanted a built-in file sender. each one shipped as a test pilot experiment for a few months. the ones that worked graduated into firefox. the ones that did not got the obituary post.

the program ran for about two and a half years and shut down on january 22 2019, with a thank-you note on the mozilla blog. the official reason was that the culture of experimentation had spread across mozilla to the point that a separate program was redundant. the cynical reading is that the 2018 reorg eliminated the team that had been running test pilot full-time. both are probably half-true.

~ the chronicle ~

~ how it ran ~

the recruitment side was clever. you installed a small extension that gave you the experiment chooser, and each experiment was its own extension that auto-installed once you opted in. mozilla collected anonymized telemetry on usage, ran short surveys inside the experiment itself, and published the results on the test pilot blog. the cycle was about three months from launch to a graduation or retirement decision. by program standards that was fast.

the audience was about a hundred thousand daily users at the peak. that is small for a firefox feature and large for a beta program. enough signal to tell mozilla whether an experiment was being used at all, not enough to predict mainstream uptake. that gap was the program's load-bearing limitation. send graduated and lasted eighteen months in public release before mozilla killed it during the 2020 restructuring.

~ what got incubated ~

the long list of test pilot experiments reads like a history of mid-2010s mozilla taste. firefox containers came out of the program and is now a stable feature in modern firefox. activity stream became the new tab page. page shot graduated as firefox screenshots. lockbox became firefox lockwise on ios before mozilla retired that too. send shipped from test pilot into a public service. tab center, min vid, snooze tabs, voice fill, notes, color, side view, all ran for a few months each.

the experiments mozilla never graduated are the more interesting list. cliqz privacy was a partnership with a german startup that turned into a small political mess about telemetry, and mozilla pulled it after about a year. firefox advance, an arxiv-style content recommender, ran for a few weeks and disappeared. these were honest failures shipped to a self-selected audience, which is the kind of failure that does not haunt a product team the way a shipped flop does.

~ the long tail ~

mozilla relaunched a smaller program in september 2019 under the same name, this time built around firefox private network and firefox monitor. that second test pilot ran for a couple years on a much narrower scope and quietly faded. the original 2016 program had been a real culture object inside mozilla. the relaunch was a brand recycling. by 2022 testpilot.firefox.com redirected to a static page and the experiment chooser was gone.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »