A new study reverses the result of an old one — just by adding fresh data.
A much-hyped 2014 study, combining more than a decade’s worth of data (1999–2010), found that medical-marijuana laws reduce opioid-painkiller overdoses. It seemed like a glimmer of hope amid a worsening epidemic. People would toke reefer instead of popping Oxy if you just gave them the chance to.
There was nothing particularly wrong with the paper. Its methods were pretty standard when it comes to studying the effects of public-policy changes. To oversimplify a bit, it compared overdose rates in states that enacted medical-marijuana laws with trends nationwide, and it also took account of some other factors — including unemployment rates and prescription-drug-monitoring programs. The upshot was that when a state had a medical-marijuana law in effect, its rate of painkiller ODs tended to be about a quarter lower than what you’d otherwise expect: an enormous improvement when a drug epidemic is killing tens of thousands of people a year. In fact, drug overdoses were climbing so quickly that the reduction, which seemed to get stronger the longer a law had been implemented, merely slowed the growth of opioid ODs in these states rather than causing a drop in absolute terms.
But this effect was a mirage, according to a new paper from Stanford’s Chelsea L. Shover and three coauthors in the Proceedings of the National Academy of Sciences. Using the same methods as the original study but adding seven years’ worth of new data, it finds that the connection between medical-marijuana laws and overdose rates didn’t merely fade out with time, but reversed. This suggests that a widely touted benefit of marijuana legalization is not real. And it should instill in us a wider skepticism toward strong claims about the effects of public policies.
The new report is in essence a time machine, showing what would have happened if the original research had been conducted later than it was. If the study had included data through 2011 or 2012, it still would have found a good effect from medical-marijuana laws. If its data had ended anywhere between 2013 and 2016, by contrast, it would have found no statistically significant effect. And when all of the data currently available are included — through 2017 — there’s a correlation between pot laws and painkiller ODs again, a positive one. And by “positive” I don’t mean good, but that medical-marijuana laws are associated with higher overdose rates: 23 percent higher, essentially a mirror image of the original finding.
The authors are highly skeptical that the effect of broadening access to pot radically changed in the past decade for some reason, or that newer pot laws somehow have the opposite effect of old ones. They check to see if different types of laws — such as legalizing recreational pot in addition to medical pot, or placing strict limits on the strength and availability of medical pot — have different effects; they find no good evidence for this, though their estimates are imprecise. Their ultimate suggestion is that the old and new results are probably both spurious, driven by unknown variables the analysis doesn’t include, and that medical marijuana doesn’t have a big enough effect on opioid abuse for it to be measured reliably. This is backed up by the fact that their new result fades, to the point of becoming statistically insignificant, when they tweak the methods a bit.
It’s possible, however, to spin a darker tale. Indeed, the study has some roots in one such narrative. In his much-derided anti-ganja book Tell Your Children (see my review), the journalist Alex Berenson had a mathematically adept friend conduct an analysis similar to the one in the current study, with the data ending in 2016, and with the result that there was no connection between pot laws and opioid ODs. That friend’s name? Sanford Gordon of New York University: a coauthor of the new study.
In interpreting Gordon’s results, Berenson posited “geographic coincidence” as the reason for the original negative correlation. As it initially spread, the opioid epidemic hit Appalachian and Midwestern states hardest, while the states passing medical-marijuana laws were disproportionately out west. Thus the negative relationship in the beginning, and thus why it faded as both the epidemic and medical-marijuana laws spread further in the 2010s.
Berenson didn’t stop there, though, offering some arguments that pot is a gateway drug (as has long been claimed by its opponents), such as the fact that people who experiment with weed often later abuse other drugs too — as well as the fact that, in Gordon’s run of the numbers, while pot laws didn’t correlate with opioid deaths, rates of pot use did. I did not find these arguments all that compelling. But Berenson has to be pretty thrilled with the new findings, and especially with the possibility that the original correlation was spurious but the new one is not.
I wouldn’t bet on this being the case, but it can’t be dismissed out of hand. And it will become harder to dismiss if the relationship strengthens further in the future and no one provides a good alternative explanation. Then again, maybe the old result will rise from the dead somehow, especially as we learn more about the changing character of the opioid epidemic. (In particular, these days it’s less about prescription meds and more about heroin and Fentanyl.) As they say, more research is needed.
The broader conclusion, of course, is that the statistical techniques we use to evaluate public policy — and by extension, many of the studies used to back up condescending claims that this or that policy is “evidence-based” and opposition to it is “anti-science” — are not as powerful as they may seem. In this case a law was associated with a roughly one-quarter decrease in painkiller overdoses in a 2014 study, and with a roughly one-quarter increase in such ODs in a 2019 study using the same, generally accepted methods. That is a swing so dramatic it would be comical if it didn’t represent so many horrifying deaths and our cluelessness about how to stop them.
For the record, I believe pot should be legal — that getting baked on weed is “well within the range of risks we should allow adults to take,” as I put it in my review of Berenson’s book. But as we weigh the costs and benefits, it seems safe to say we can strike “reduces opioid overdoses by a quarter” from the benefits side, and turn a more skeptical eye to such big claims in the future.