Inktomi

» see inktomi.com as it lived, on the wayback machine

~ the obituary ~

inktomi was the engine under everything i used and did not know the name of. by '98 i was on a 14.4k modem in turkey hitting hotbot once in a while when altavista returned junk. hotbot was inktomi. msn search was inktomi by '99, anyway. yahoo's web results were inktomi from june 2000 until google won the contract a few weeks later. for a stretch of two or three years almost every search engine that was not altavista was actually a different paint job on top of the same berkeley spider index. that fact only became visible to me much later.

eric brewer was a berkeley cs professor working on parallel-systems research. paul gauthier was his graduate student. they figured out in '95 that you could crawl and index the web by lashing together a couple hundred sun workstations instead of paying for one giant smp box like altavista's dec alpha cluster. the architecture paper went out in '96. the company spun out the same year. brewer kept the academic post, gauthier ran the engineering side. the name came from a lakota spider trickster, iktomi, picked because the spider crawls a web. that level of literary care for the brand was unusual for a 1996 search company.

the first customer was hotbot, a wired magazine joint venture that launched in may '96 as a flagship for the inktomi backend. hotbot's color-saturated ui and the inktomi index underneath it pulled the technical-press attention away from altavista for a stretch in '97 and '98. the index was bigger than altavista's, the crawl was fresher, the ranking less spammy. wired turned hotbot into a recognizable consumer brand without anyone in the press caring much about who was running the spiders.

inktomi figured out it could license that index to other portals that did not want to build their own search team. msn licensed it. iwon and looksmart bought results from it during the same window, with aol and several smaller portals stacking on later. by '99 the b2b search-licensing market was inktomi's invention and inktomi was the dominant supplier. yahoo, who had been pairing its human-edited directory with altavista results, switched to inktomi in june 2000. that was the high point. it lasted seven weeks. google won the yahoo deal in early july and the licensing economy started to slide.

~ the case file ~

~ what it actually was ~

the technical contribution was real. brewer's lab paper on parallel search across commodity hardware was one of the building blocks that the google papers later cited (roughly). honestly, inktomi's traffic server, the proxy/cache product they built on the side, was donated to apache in 2009 by yahoo. it still ships as apache traffic server today. the index architecture that everyone used through the late '90s was inktomi's design even after the brand was gone.

the consumer-facing failure was structural. inktomi never had a destination site of its own. hotbot was a wired property, not an inktomi property. msn search was a microsoft property. yahoo was yahoo. when google launched a clean front page in '99 with its own brand on the door, the licensing customers did not just lose a vendor. they lost the entire premise that the search-result layer was a commodity you could rent.

~ the slide ~

the july 2000 yahoo loss to google was the visible inflection. the actual cracks had opened earlier. inktomi's revenue mix was heavily b2b licensing, which carried thin margins compared to google's emerging adwords money. the company's ipo had peaked at 241 dollars a share in march 2000 alongside the rest of the dotcom run. by november 2002 the stock was under two dollars and the company had cut headcount to about 200 people. the dotcom collapse took the b2b portals' marketing budgets with it. nobody was paying for licensed search results when their own pageviews were collapsing.

yahoo announced the acquisition march 19 '03 for 235-ish million, finalized at about 241 million. the price was a hair below the all-time stock peak per share, which by march 2003 was a sad little detail of timing. yahoo wanted the index technology for the inhouse search stack that would become yahoo search by '04. the inktomi brand was wound down quietly. brewer went back to berkeley fulltime.

~ what carried over ~

the practical inheritance was the proof that distributed crawling on commodity machines worked. google's architecture was a different and better implementation of the same instinct (or whatever it was called). by the time yahoo's inhouse stack was running on the post-inktomi codebase in '04, google had already won the consumer search market and was about to start winning the licensing market too. yahoo would re-license search results from microsoft bing by '09, completing the loop where the company that bought inktomi to escape google ended up renting from google's only major competitor.

inktomi gets remembered now mostly in two places. the original brewer lab paper from '96 is still cited in distributed-systems coursework, the cube logo gets recognized by people who used hotbot in the late '90s and only later learned what was actually running underneath. for an engineering team that briefly powered the majority of english-language web searches, the cultural memory is thinner than it should be. the back end never gets the credit.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »