JournalSpace

» see journalspace.com as it lived, on the wayback machine

~ the obituary ~

journalspace's peak years overlap with my university window. the platform did not. it sat in the english-language web's middle tier of blogger and livejournal and xanga and typepad, the smaller of the small, a few hundred thousand accounts at most. honestly, the founders ran it as a side operation out of los angeles starting 2002. the homepage looked like a clean version of livejournal with friends pages and comment threads and inline rss. people kept journals there for six or seven years without anything dramatic happening on the operational side.

the death event happened the last week of 2008. somebody at journalspace, suspected later to be a fired it administrator with a grudge, overwrote the entire production database with random data on the morning of december 22. the server then dutifully replicated the destruction across to the raid mirror, which had been configured to be the only backup mechanism. there was no off-site backup, no nightly dump, no tape rotation. the mirror was the backup, and the mirror did its job. on the wipe morning that meant the mirror copy was now identical to the wiped main.

the drives were pulled out and shipped to drivesavers in northern california. honestly, the recovery quote came back negative. random-data overwrite at the block level beats forensic recovery. on january 2 2009 the journalspace homepage posted a short notice that the platform was dead and the users had lost their archives. the domain was sold on january 11 to new owners who relaunched a blog site at the same url, without the old data. the original community was gone in a calendar week.

~ the case file ~

~ the day it died ~

this part you had to be inside the operations team to feel. the company's own postmortem went up a few days after the wipe and was unusually candid for the era. the it administrator in question had been caught stealing from the company earlier and had been let go. on the way out, the company believed, he wiped the primary database. the raid configuration had been his architecture. he had set up automated backups for the http tier with the php code, then chose to rely on the raid mirror as the only protection for the sql tier. the wipe blast-radius landed exactly where the missing backup was.

the postmortem also pointed out that the same admin used to talk frequently about how smart he was. you can hear the bitterness in the announcement post even fifteen years later. honestly, the raid disks were sent to drivesavers on december 22. the recovery effort ran for about a week before drivesavers confirmed the random-data overwrite had no residual signal to work with. by new year's eve the company knew the site was unrecoverable, the announcement was timed for january 2 to give users a clean cutoff.

~ the funny part ~

journalspace became the standing raid-is-not-a-backup case study for the next decade of sysadmin training material. backblaze published a long writeup. brent ozar published a database-administration version. honestly, the disaster-recovery community used the journalspace case as the standing example of why you need at least one copy of the data not in the same building as the production server. the irony of the company's name only got more visible after the wipe. there was no journal, there was no space.

~ what got buried with it ~

this part i was not in. the users were. a sample of bloggers retrieved fragments through google's cached pages in the days after the wipe, then through archive.org snapshots taken before december. the actual private posts, the friends-only entries, the unlinked drafts, the dialogue under comment threads, none of that was in google's cache or in archive.org. that material was simply gone.

in those days. the domain has been resold several times since. the current site at journalspace.com has nothing to do with the 2002-2009 community. that community is one of the small permanent absences in the english-speaking web's history.

@}-->--

              )
             (_)
            .-'-.
            |   |
            |   |
            |   |
            |   |
          __|   |__   .-.
       .-'  |   |  `-:   :
      :     `---'     :-'
       `-._       _.-'
           '""""""

« previous  ·  random  ·  next »