Microsoft Photosynth

» see photosynth.net as it lived, on the wayback machine

~ the obituary ~

photosynth was the rare microsoft product that came out of a research lab and felt like a glimpse of the future. blaise aguera y arcas demoed it at TED in 2007, before the consumer release: take a folder of photos of trinity college library or notre dame, run a structure-from-motion algorithm on them, and the software figured out where each camera had been standing and assembled the photos into a 3D point cloud you could fly through. i did not use it as a tourist. i was already a computer engineering student in turkey reading about exactly this kind of vision algorithm, and the photosynth demo was the moment the math got demoted from paper to download.

the technology came out of noah snavely's phd work at the university of washington, originally as a project called photo tourism. microsoft live labs picked it up after the company acquired seadragon software in 2006, and blaise aguera y arcas, who had built seadragon's deep-zoom tech, directed the project inside live labs. the consumer version launched in august 2008 as a free web service with a windows uploader client. you took 30 to 200 photos of a place, uploaded them, and the photosynth servers ran the bundle adjustment and the feature-matching pipeline. the service handed you back a 3D scene you could share with a link.

in march 2010 microsoft added a panorama mode based on microsoft image composite editor, which let you stitch a single high-resolution panorama without the full 3D rebuild. that pivot was where photosynth's mass-market moment briefly arrived. the panorama mode was easier and the output looked better on cheaper hardware. but microsoft never built it into windows itself or shipped it as a real phone app outside of a brief windows-phone pilot, and the panorama mode never became something most people kept open. the website went down on february 6 2017. the underlying tech got folded into microsoft pix on the iphone later that year, then folded again into microsoft flight simulator 2020 to help generate the world.

~ the case file ~

~ what it was trying to do ~

structure-from-motion was a research-grade problem in 2007. the pipeline needed feature descriptors that matched across photos shot from different angles, plus bundle adjustment that could solve the camera-position equations across hundreds of images without exploding. it also needed enough server compute to run the whole thing in a reasonable time. photosynth packaged that pipeline into a download. the academic version of this had been running in computer vision labs since the late 1990s. the consumer version showed up in 2008, ten years late by research time, but still ahead of where any phone app of the era could go.

~ the next chapter ~

what photosynth was an early version of, every smartphone now does. apple's panorama mode in ios 6 was a direct continuation of the panorama side. google street view and apple flyover use larger versions of the same camera-pose-recovery math, and so do the consumer ar-measure tools and the lidar 3D scanning on newer phones. the drone-footage 3D-rebuild that civilian mapping companies sell to municipalities is the same lineage shrunk down to a different sensor stack. i now work on edge vision systems for uavs, which is, roughly, photosynth's pipeline scaled down to run on a flight controller.

~ the long tail ~

the website is gone. the.synth files that several million people uploaded sit in microsoft's archives, unrenderable, since the silverlight viewer they needed died with the broader silverlight retirement. the panorama exports survived as static jpgs. blaise aguera y arcas is now vp at google research and works on the foundation models side. the algorithms photosynth used are open knowledge now and ship inside opencv. the lab demo from 2007 still circulates on youtube every few years when someone rediscovers it and treats it as a piece of pre-iphone vision history.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »