Much as I did Tuesday, I’d like to pop in with a quick word about the Institute for Health Metrics and Evaluation COVID-19 model heavily relied on by the White House. Once again, Tobias Hoonhout and Andy McCarthy got there first.
My last post explained the first downgrade this week; this one will focus on the second. Yes, it’s highly frustrating to have two sizable revisions, in the same direction, just days apart, but here’s an attempt to piece together what happened and why.
What the model aims to do is draw a curve representing the epidemic. It rises exponentially for a while, then levels off and falls. The model uses existing data from the U.S. to draw the beginning of the curve (i.e., the past), and then guesses what the rest of the curve will look like based on how things have played out in other countries.
Unlike the previous revision this week, the most recent one didn’t come with a detailed explanation. But apparently some new data came in better than expected, which IHME finally got around to stating publicly a day later:
New data from several states indicates a decline in the overall deaths we predict for the US. It’s important to note that our forecasts range from 126,703 to 31,221 deaths. As data comes in, our estimates will change, much like weather forecasts adjust. https://t.co/wk0u3TugF6 pic.twitter.com/9yw5uXIRoI
— Institute for Health Metrics and Evaluation (IHME) (@IHME_UW) April 9, 2020
So, why did the model not see that these lower numbers were coming? It has always factored in states’ social-distancing policies — and even assumed that states without such policies would implement them soon — so it’s not a case of these policies’ having their expected effect.
One way to spin it is that social distancing is more effective than anticipated, perhaps because a lot of it happened before governments mandated it:
Our estimates for total COVID-19 deaths have dropped from 81,766 to 60,415 in today’s update. Why? Because social distancing measures in the US are working. Some Americans stayed at home before the orders came. Encouraging news #COVID19 #socialdistancing »https://t.co/KIhdSrXK7z
— Ali H. Mokdad (@AliHMokdad) April 8, 2020
Another, of course, is that the model was or is just off. That would hardly be surprising for an attempt to predict what will happen in a situation as chaotic and unclear as this one, and there have been some solid criticisms of the way the model works (many of which actually say it’s too optimistic). Here, for example, is a thread laying out how the model can behave in volatile ways when small bits of new information are added:
1. Models have their strength and weaknesses. It's valuable to understand both. While the @IHME_UW model has certain advantages over other approaches, I want to focus here on a disadvantage, namely the absence of an underlying mechanistic / bottom up / process-based framework.
— Carl T. Bergstrom (@CT_Bergstrom) April 7, 2020
As I’ve said from the beginning, when I first took note of the effort from Imperial College, any model trying to do what these models do is going to be wrong to some extent. I wouldn’t take IHME’s work as gospel, especially given all the volatility in such a short time. (The model doesn’t take itself as gospel, for that matter, given the huge uncertainty ranges it provides.) At most it can give us an informed guess that slowly gets better as new information comes in. If I were managing a hospital or governing a state, I’d definitely check what it says, but I’d consider it alongside other sources.
A few final points.
First, if this model is wrong, it can be wrong in either direction. I’m not aware of any model that says we should just go ahead and reopen, and the experiences of China, Italy, and New York City would suggest it’s unwise.
Second, and relatedly, the model is only of the first wave of this epidemic. It does not try to predict any resurgence that could happen when measures are lifted, whether that’s now or after the first wave is under control.
Third, when something grows exponentially until it’s squelched and rapidly fades out — when it’s measured by how quickly it doubles or is cut in half — revisions that would seem big in other contexts are nearly unavoidable.
Lastly, if the epidemic ends up killing the current prediction of 60,000 Americans in just a few months with the lockdowns, what does that say about what would have happened if we’d just let it rip through the population?