And it’s a) not what Team Obama is telling us and b) not a pretty picture. David Auerbach writes:
Err Engine Down
What really went wrong with healthcare.gov?
Of all the terrible websites I’ve seen, healthcare.gov ranks somewhere in the middle. It has been difficult if not impossible to sign up, and customer service has been inadequate. But it’s certainly better than the NYC Department of Education site that I attempted to help a friend navigate two years ago, in hopes of her getting paid her actual salary instead of a default salary; the blatantly inept Web code got the best of us. And it’s better than the evanescent Web encyclopedia Cpedia, which rolled out with pages that literally consisted of nonsense (such as “Clickbooth Cuil but not avail due to flooding traffics and making their servers ‘too hot’ to handle”). The problems plaguing healthcare.gov aren’t due to a unique coding failure or a unique government failure—plenty of products have had similar early deficits, such as the Electronic Arts server bugs that rendered SimCity unplayable by most for more than a week after it was released this March. So healthcare.gov’s failures are not uncommon—they’re just exceptionally high-profile.
The Redditors picking apart the client code have found some genuine issues with it, but healthcare.gov’s biggest problems are most likely not in the front-end code of the site’s Web pages, but in the back-end, server-side code that handles—or doesn’t handle—the registration process, which no one can see. Consequently, I would be skeptical of any outside claim to have identified the problem with the site. Bugs rarely manifest in obvious forms, often cascading and metamorphizing into seemingly different issues entirely, and one visible bug usually masks others.
There are a few clues, however. The site’s front end (the actual Web pages and bits of script) doesn’t look too bad, but it is not coping well with whatever scaling issues the back end (account storage, database lookups, etc.) is having. I tried to sign up for the federal marketplace six days after rollout. The site claimed to be working, but after I started the registration process, I sat on a “Please Wait” page for 10 minutes before being redirected to an error page:
“Sorry, we can’t find that page on HealthCare.gov.”
Except that wasn’t the problem, since the error message immediately below read:
“Error from: https://www.healthcare.gov/oberr.cgi?status%3D500%20errmsg%3DErrEngineDo….”
To translate, that’s an Oracle database complaining that it can’t do a signup because its “engine” server is down. So you can see Web pages with text and pictures, but the actual meat-and-potatoes account signup “engine” of the site was offline. A good site would have translated that error into a more helpful error message, such as “The system is temporarily down,” or “President Obama personally apologizes to you for this engine failure.” But it didn’t.
This failure points to the fundamental cause of the larger failure, which is the end-to-end process. That is, the front-end static website and the back-end servers (and possibly some dynamic components of the Web pages) were developed by two different contractors. Coordination between them appears to have been nonexistent, or else front-end architect Development Seed never would have given this interviewto the Atlantic a few months back, in which they embrace open-source and envision a new world of government agencies sharing code with one another. (It didn’t work out, apparently.) Development Seed now seems to be struggling to distance themselves from the site’s problems, having realized that however good their work was, the site will be judged in its totality, not piecemeal. Back-end developers CGI Federal, who were awarded a much larger contract in 2010 for federal health care tech, have made themselves rather scarce, providing no spokespeople at all to reporters. Their source code isn’t available anywhere, though I would dearly love to take a gander (and so would Reddit). I fear the worst, given that CGI is also being accused of screwing up Vermont’s health care website.
So we had (at least) two sets of contracted developers, apparently in isolation from each other, working on two pieces of a system that had to run together perfectly. Anyone in software engineering will tell you that cross-group coordination is one of the hardest things to get right, and also one of the most crucial, because while programmers are great at testing their own code, testing that their code works with everybody else’s code is much more difficult.
Look at it another way: Even if scale testing is done, that involves seeing what happens when a site is overrun. The poor, confusing error handling indicates that there was no ownership of the end-to-end experience—no one tasked with making sure everything worked together and at full capacity, not just in isolated tests. (I can’t even figure out who was supposed to own it.) No end-to-end ownership means that questions like “What is the user experience if the back-end gets overloaded or has such-and-such an error?” are never asked, because they cannot be answered by either group in isolation. Writing in Medium in defense of Development Seed, technologist and contractor CTO Adam Becker complains of “layers upon layers of contractors, a high ratio of project managers to programmers, and a severe lack of technical ownership.” Sounds right to me.
Likewise, the bugs around username and password standards—for example, the fact that the username required a number but the user interface didn’t tell the user about it—are not problems of scale. They’re problems of poor cross-group communication. I’d bet that plenty of people knew what was going to happen when the site rolled out, but none of them were in a position to mitigate the damage.
The rest here.





Comments