I’ve had occasion before to comment on the defects and limitations of quantitative political science, which often seems to careen from either proving the trivial or obvious, to marginalizing itself into complete irrelevance. But every so often the “quants” perform some useful self-examination, and one such case is on display in the latest issue of PS: Political Science & Politics, one of the academic journals you receive as a member of the American Political Science Association. (It’s one of those things I do so that you don’t have to.)
David Brady, Morris Fiorina, and Arjun Wilkins of Stanford University (Brady and Fiorina are also Hoover Institution fellows) have written an interesting article entitled “The 2010 Elections: Why Did Political Science Forecasts Go Awry?”, which takes note of the fact that all of the most heralded predictive models badly underestimated Republican gains in the 2010 mid-term election. The worst performing of the models, from Michael Lewis-Beck and Charles Tien, concluded Republicans would net 22 House seats. Another model, from Joseph Bafumi, Robert Erikson, and Chris Wlezien, said Republicans could gain as many as 51 seats, but the authors went on to heavily discount their own results in a passage that yields hilarity today:
The tilting toward the out-party may thus have been realized earlier than in the past. [My translation: Republicans peaked too early.] It is also possible that an unusually conservative Republican campaign could dampen the expected shift toward the Republicans. If either of these possibilities is true and the polls remain unchanged through November, the Democrats actually could have the edge come Election Day.
And Lewis-Beck and Tien concluded: “We believe this extreme scenario [a 35 seat GOP pickup] to be very unlikely to occur. In sum, the Democrats should keep their House majority.” Oh-kay.
Of course, all of this is what you might call “judgment,” not to say bias, and not driven by the data. As Brady et al. comment: “[I]t is clear from the commentary accompanying the forecasts that most of the authors were more concerned about overpredicting the Democratic losses than underpredicting them.” Brady and his co-authors then fiddle with the models to evaluate the effect of House votes for the health care bill and the Waxman-Markey cap-and-trade bill, and conclude that 40 House Democrats who voted Yes on both might well have kept their seats if they had voted No on both — enough to have saved the Democratic House majority. The authors include a number of sensible caveats about the defects of counterfactual forecasting, but can’t escape the general conclusion that “the real problem for many House Democrats was most likely the Democratic agenda.”
This won’t go over well at the next APSA convention. (Which, by the way, was moved from San Francisco to Seattle this year because a single union in SF is threatening to go on strike this summer. The APSA’s move cost the organization thousands of dollars in non-refundable deposits. I guess we shouldn’t be surprised that academic political scientists are prone to political correctness.)