The Corner

How Narwhal Beat Orca

The Atlantic has the long but engaging and amusing tale of the Obama campaign’s “Tech” team, who designed all the various apps that Obama 2012 used to fundraise, organize, contact voters, etc. There’s been plenty of controversy over the failure of the Romney campaign’s parallel efforts; much of the digital infrastructure for the campaign’s “Orca app” (the orca is the only known predator of the narwhal, the codename for Obama’s efforts) collapsed on Election Day, with Internet service going down, the app being overloaded and unready to handle full demand, lack of telephone communications, etc. The contrast with the Obama campaign’s work could not be more dramatic:

The Obama campaign’s technologists were tense and tired. It was game day and everything was going wrong.

Josh Thayer, the lead engineer of Narwhal, had just been informed that they’d lost another one of the services powering their software. That was bad: Narwhal was the code name for the data platform that underpinned the campaign and let it track voters and volunteers. If it broke, so would everything else.

They were talking with people at Amazon Web Services, but all they knew was that they had packet loss. Earlier that day, they lost their databases, their East Coast servers, and their memcache clusters. Thayer was ready to kill Nick Hatch, a DevOps engineer who was the official bearer of bad news. Another of their vendors, PalominoDB, was fixing databases, but needed to rebuild the replicas. It was going to take time, Hatch said. They didn’t have time.

They’d been working 14-hour days, six or seven days a week, trying to reelect the president, and now everything had been broken at just the wrong time. It was like someone had written a Murphy’s Law algorithm and deployed it at scale.

And that was the point. “Game day” was October 21. The election was still 17 days away, and this was a live action role playing (LARPing!) exercise that the campaign’s chief technology officer, Harper Reed, was inflicting on his team. “We worked through every possible disaster situation,” Reed said. “We did three actual all-day sessions of destroying everything we had built.”

The story also recounts how the team, assembled in the early summer of 2011, underdelivered for months, and much of their work didn’t come to fruition, and become usable by voters, until the spring of 2012. Eventually, though, the world-class team produced a range of products that gave the campaign and its volunteers a range of abilities it had never had before. It’s also a fascinating account of how the precepts of digital-development and start-up management applied, or didn’t apply, to the Obama campaign’s efforts, which were so much more vast than the Romney campaign’s.

In short, though, we never really could have hoped that the GOP would match the quality of Obama’s digital efforts, since that would require Mitt Romney hiring a guy like this:

That’s Obama for America’s tech director, Harper Reed:

If you’ve spent a lot of time around tech people, around Burning Man devotees, around startups, around San Francisco, around BBSs, around Reddit, Harper Reed probably makes sense to you. He’s a cool hacker. He gets profiled by Mother Jones even though he couldn’t talk with Tim Murphy, their reporter. He supports open source. He likes Japan. He says f*** a lot.  He goes to hipster bars that serve vegan Mexican food, and where a quarter of the staff and clientele have mustaches. . . .

Reed has friends like the manager of the hip-hop club Empire who, when we walk into the place early on the Friday after the election, says, “Let me grab you a shot.” Surprisingly, Harper Reed is a chilled vodka kind of guy.

Patrick Brennan was a senior communications official at the Department of Health and Human Services during the Trump administration and is former opinion editor of National Review Online.


The Latest