Over at The New York Times today, we get a nice little window into how social science sometimes works, and why it often mixes poorly with politics.
Times reporter Margot Sanger-Katz turns her attention to an academic paper that helped make Elizabeth Warren a prominent figure in progressive politics—and eventually of course a senator from Massachusetts. The paper, which Warren co-authored with three colleagues in 2005, claimed that more than 40 percent of bankruptcies in America were caused by medical costs.
The startling statistic soon made headlines, though it also immediately attracted criticism. As Sanger-Katz writes:
But from the beginning many economists questioned the paper’s approach, which relied on surveys of nearly 1,800 Americans who had declared bankruptcy in 2001 and interviews with about half of them on their views about the causes of their financial woes after the fact. The researchers counted a bankruptcy as due to injury or illness if a person had a medical debt of more than $1,000; said illness or injury caused a bankruptcy; missed more than two weeks of work because of illness; or mortgaged a home to pay medical bills.
That is obviously a very broad and varied definition, intended to maximize the resulting figures. Sanger-Katz quotes Craig Garthwaite, a health economist at Northwestern, as saying: “There are no reputable economists who I deal with who believe the number in the paper or the methods in the paper are appropriate in trying to get at the true underlying question.”
That hasn’t always been clear to people in the political and policy worlds, though. In 2006, when I worked as a health policy staffer at the White House, President Bush’s speechwriting team wanted to cite the figure in a passage on health care in that year’s State of the Union address and were warned away from it in no uncertain terms by an economist serving on the Council of Economic Advisers (who has since gone on to do important work at Harvard and lately the University of Chicago). She dismissed the figure out of hand and harshly criticized the paper’s methodology.
But the prominence of the paper and its claims only grew. When Warren and her co-authors updated their work in 2009, they claimed an even larger number, concluding that 62 percent of American bankruptcies were essentially medical bankruptcies. That figure played a prominent role in the case for Obamacare that year and the following.
None of that did much to persuade economists and other researchers, though. And in March of this year, a team of economists from M.I.T., Northwestern, and UC Santa Cruz published a paper that tried to quantify medical bankruptcies using more direct evidence and concluded that only four percent of bankruptcies could be attributed to medical costs.
As Sanger-Katz deadpans, “There is, of course, a huge difference between 4 percent and 62 percent.” Of course.
This newer paper isn’t the last word either, needless to say. The research will continue, and controversies about methodology are one of the ways such research progresses. But this story does help us see how the politicization of social-science research can distort both policy debates and academic work—even as it launches some political careers.