Google Code Search

» see google.com/codesearch as it lived, on the wayback machine

~ the obit ~

Google Code Search went up in Labs on 5 October 2006 and did one thing nobody else was doing. It crawled public source code, tarballs and zip files and CVS and Subversion and git repositories, indexed the text of it, and let you search it with a regular expression. An actual POSIX pattern in the box, with the operators lang: and license: and file: on top to narrow the results down.

If you learned to program in that window you learned by reading, and the reading material was whatever tarball you could find. The alternatives were a forum thread from three years earlier or a man page that described the function without ever showing you somebody using it. Code Search let you ask the more useful question directly: show me every place in public code where this call appears next to that one. It was a teaching tool disguised as a search engine, and it was free, and it lasted five years.

It stayed a Labs product for its entire life, which in 2011 turned out to matter.

~ the rap sheet ~

~ how it worked ~

The hard part was never the crawling. It was that regular expressions and inverted indexes do not naturally get along: a search index answers questions about words, and a pattern like [a-z]+_key is not a word. The trick, worked out by Russ Cox and his colleagues, was to translate the expression into a query over three-character sequences. The trigram index threw away every document that could not possibly match, and the real matcher then ran over the small pile that was left.

The matcher was RE2, written so that a pathological pattern could not take the server down, which matters when the pattern is coming from a stranger on the internet. Google open sourced RE2 in March 2010 and it went on to sit inside a large amount of other software. In January 2012, the same month the service died, Cox published a full write-up of how the thing had worked and released a standalone version you could point at a directory on your own disk. The product was deleted and the method was written down, which is more than most shutdowns leave behind.

~ the week it scared everyone ~

Within days of launch, security people worked out what a regex over all public code is also good for. Chris Shiflett was posting example queries for injection-prone patterns inside a week, and Bruce Schneier wrote it up the same month. Searches aimed at config files with credentials in them were returning results, and Google started filtering some of those queries almost immediately.

The argument that followed is one the industry has had many times since, most recently about the model that suggests code and the one that reads your repository. A tool that lets a defender audit the world's published source lets an attacker do the same reading, and Google spent the rest of the product's life tuning which queries it would answer.

~ the fall sweep ~

On 14 October 2011 Bradley Horowitz published "A fall sweep", one of a series of posts through Larry Page's first year back as chief executive in which Google listed what it was closing. Code Search was on it along with the API, with a final date of 15 January 2012. Jaiku went in the same round.

A cut-down search limited to projects hosted on Google Code itself limped along for another year or so, and then that went too when Google Code was wound down in 2015. After that there was a real gap. Krugle and Ohloh covered fragments of it, none of them at the scale of a Google crawl, and it took until GitHub shipped its rebuilt code search in the 2020s for anything comparable to exist again. More than a decade for a feature to come back, and the original one was switched off because it did not fit a slide about focus.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »