A recent “blockbuster” study in Science turned out to be a fraud. But there’s a more pervasive problem with academic journals these days, the question of replication.
The scientific method requires that a study be capable of being replicated for it to be valid. If it can’t, then something is wrong, and the hypothesis is questionable. For decades, some journals have at least recognized that underlying data and methodology for studies should be available, both for peer reviewers and so that researchers can replicate the study.
But replication is unusual. In the latest issue of Econ Journal Watch, three authors report on their survey of replication studies in economics journals.
Maren Duvendack, Richard W. Palmer-Jones, and W. Robert Reed report that while such studies have been increasing since 2000, they are still relatively rare, for several reasons. Replicating a study is considered unoriginal; merely confirming the findings of others is unlikely to advance a researcher’s career. Alternatively, if the replication disputes the original findings, it may attract attention—but also create disputes with powerful people in the field. Nor do editors like mere replications—the top economics journal, the American Economic Review, has never published a study that simply confirms the findings of a previous study.
Duvendack, Palmer-Jones, and Reed estimate that, based on the Journal of Applied Economics, which does not appear to discriminate between confirmations and disputes, that about 65 percent of published replication studies cannot confirm the original findings, in all or in part. They aren’t sure whether that is because only failure-to-confirm studies are attractive to researchers and editors or whether it signifies a high level of poor studies.
Econ Journal Watch is a peer-reviewed online journal that critiques economic studies and promotes classical liberalism. (I’m an editorial adviser.)