One of the lessons we should draw from Hurricane Irma is one we ought to have learned by now: not to treat predictions of the future that are based on complex systems as facts.
We do lots of predicting these days based on reams of data fed into mathematical models, and hurricane prediction is among the best and most sophisticated of these. Hurricane models are based on scores of prior observations, and there’s a lot riding on their ability to keep people up to date on the path of a storm. The more hurricanes we observe and match to the models, the more accurate they can become, and by historical standards, they’re very good at it.
And yet, most hurricane modeling is quite bad at long-term forecasting, even over a horizon of a week; the models need to be continuously updated, and their predictions revised hourly based on new data, as each storm progresses (even once Irma’s size and path were generally foreseeable, forecasters didn’t know for days if the storm’s eye would head up Florida’s Atlantic or Gulf Coasts). And even then, the best of models can be off in very important ways due to the interaction of the known elements of a complex system (weather over the water). That’s exactly what happened with Irma as it veered away at the last minute from delivering the kind of catastrophic damage to Florida’s mainland that it dropped on the Keys and several of the islands to the south. As Bloomberg reports:
Twenty miles may have made a $150 billion difference. Estimates for the damage Hurricane Irma would inflict on Florida kept mounting as it made its devastating sweep across the Caribbean. It was poised to be the costliest U.S. storm on record. Then something called the Bermuda High intervened and tripped it up…If Irma had passed 20 miles west of Marco Island instead of striking it on Sunday, “the damage would have been astronomical.” A track like that would have placed the powerful, eastern eye wall of Irma on Florida’s Gulf Coast. By one estimate, the total cost dropped to about $50 billion Monday from $200 billion over the weekend. The state escaped the worst because Irma’s eye shifted away from the biggest population center of Miami-Dade County. The credit goes to the Bermuda High, which acts like a sort of traffic cop for the tropical North Atlantic Ocean. The circular system hovering over Bermuda jostled Irma onto northern Cuba Saturday, where being over land sapped it of some power, and then around the tip of the Florida peninsula, cutting down on storm surge damage on both coasts of the state.
This was not a failure of the models, so much as an inherently uncertain event that overwhelmed the predictive capacity of the models:
For 10 days, computer-forecast models had struggled with how the high was going to push Irma around and when it was going to stop, said Peter Sousounis, director of meteorology at AIR Worldwide. “I have never watched a forecast more carefully than Irma. I was very surprised not by how one model was going back and forth — but by how all the models were going back and forth.”…Now meteorologists are watching Hurricane Jose churn in a circle north of the Leeward Islands. Sousounis said computer models are struggling to predict whether it will pass harmlessly out to sea or strike Cape Cod at the end of Massachusetts. Jose won’t give up the answer for more than a week.
While there may be lessons in these storms to incorporate into future improvements in the models, the real lesson here is much the same as the lesson after election forecasters like Nate Silver had Donald Trump with about a 1 in 3 chance to win the 2016 election based on the available polling entering Election Day: probabilities aren’t facts, there are limits to our ability to predict complex systems, and an event with a nonzero projected chance of happening will sometimes happen. The more complex the system, the more likely it is to thwart efforts at projection. And yet, we keep seeing predictions about vastly more complex systems like the climate or the economy being treated as if they were hard, undebatable facts – even when the predictors have a track record of recurring failure of their past predictions, rather than the records of Silver or the hurricane modelers, who have been right more than wrong. Government projections of future revenues from tax legislation, or the cost of federal programs, are rarely correct. The CBO has been wrong about Obamacare enrollment continuously for years, yet its projections are treated as if they were the scores from yesterday’s ballgames. The 2008 financial crisis was in large part a story of systemic failures, in the private and public sectors, to project the trajectory of housing markets and their impact on financing structures.
In the words of Yoda, “always in motion, the future is.” The world is a complicated place, and it is often only in hindsight that we see how all the pieces interact. Both history and data are useful tools for understanding where we may be headed at any given moment, and indeed, it was prudent for people on the Florida coast to take seriously the threat that Irma might have been a lot worse (it was plenty dangerous enough as is). But we should all have a little more humility and a lot less hubris and scientistic triumphalism about “facts” that are really just educated guesses with numbers. This won’t be the last time they are this far off.