How Irritated Should We Be with IHME Right Now?

Much as I did Tuesday, I’d like to pop in with a quick word about the Institute for Health Metrics and Evaluation COVID-19 model heavily relied on by the White House. Once again, Tobias Hoonhout and Andy McCarthy got there first.

My last post explained the first downgrade this week; this one will focus on the second. Yes, it’s highly frustrating to have two sizable revisions, in the same direction, just days apart, but here’s an attempt to piece together what happened and why.

What the model aims to do is draw a curve representing the epidemic. It rises exponentially for a while, then levels off and falls. The model uses existing data from the U.S. to draw the beginning of the curve (i.e., the past), and then guesses what the rest of the curve will look like based on how things have played out in other countries.

Unlike the previous revision this week, the most recent one didn’t come with a detailed explanation. But apparently some new data came in better than expected, which IHME finally got around to stating publicly a day later:

So, why did the model not see that these lower numbers were coming? It has always factored in states’ social-distancing policies — and even assumed that states without such policies would implement them soon — so it’s not a case of these policies’ having their expected effect.

One way to spin it is that social distancing is more effective than anticipated, perhaps because a lot of it happened before governments mandated it:

Another, of course, is that the model was or is just off. That would hardly be surprising for an attempt to predict what will happen in a situation as chaotic and unclear as this one, and there have been some solid criticisms of the way the model works (many of which actually say it’s too optimistic). Here, for example, is a thread laying out how the model can behave in volatile ways when small bits of new information are added:

As I’ve said from the beginning, when I first took note of the effort from Imperial College, any model trying to do what these models do is going to be wrong to some extent. I wouldn’t take IHME’s work as gospel, especially given all the volatility in such a short time. (The model doesn’t take itself as gospel, for that matter, given the huge uncertainty ranges it provides.) At most it can give us an informed guess that slowly gets better as new information comes in. If I were managing a hospital or governing a state, I’d definitely check what it says, but I’d consider it alongside other sources.

A few final points.

First, if this model is wrong, it can be wrong in either direction. I’m not aware of any model that says we should just go ahead and reopen, and the experiences of China, Italy, and New York City would suggest it’s unwise.

Second, and relatedly, the model is only of the first wave of this epidemic. It does not try to predict any resurgence that could happen when measures are lifted, whether that’s now or after the first wave is under control.

Third, when something grows exponentially until it’s squelched and rapidly fades out — when it’s measured by how quickly it doubles or is cut in half — revisions that would seem big in other contexts are nearly unavoidable. 

Lastly, if the epidemic ends up killing the current prediction of 60,000 Americans in just a few months with the lockdowns, what does that say about what would have happened if we’d just let it rip through the population?