Phi Beta Cons

How Many Academic Studies Can Be Replicated?

A review of economics journals suggests a high percentage of irreproducible studies

A recent “blockbuster” study in Science turned out to be a fraud. But there’s a more pervasive problem with academic journals these days, the question of replication.

The scientific method requires that a study be capable of being replicated for it to be valid. If it can’t, then something is wrong, and the hypothesis is questionable. For decades, some journals have at least recognized that underlying data and methodology for studies should be available, both for peer reviewers and so that researchers can replicate the study.

But replication is unusual. In the latest issue of Econ Journal Watch, three authors report on their survey of replication studies in economics journals.

Maren Duvendack, Richard W. Palmer-Jones, and W. Robert Reed report that while such studies have been increasing since 2000, they are still relatively rare, for several reasons. Replicating a study is considered unoriginal; merely confirming the findings of others is unlikely to advance a researcher’s career. Alternatively, if the replication disputes the original findings, it may attract attention—but also create disputes with powerful people in the field. Nor do editors like mere replications—the top economics journal, the American Economic Review, has never published a study that simply confirms the findings of a previous study.

Duvendack, Palmer-Jones, and Reed estimate that, based on the Journal of Applied Economics, which does not appear to discriminate between confirmations and disputes, that about 65 percent of published replication studies cannot confirm the original findings, in all or in part. They aren’t sure whether that is because only failure-to-confirm studies are attractive to researchers and editors or whether it signifies a high level of poor studies.

Econ Journal Watch is a peer-reviewed online journal that critiques economic studies and promotes classical liberalism. (I’m an editorial adviser.)

Jane S. ShawJane S. Shaw retired as president of the John W. Pope Center for Higher Education Policy in 2015. Before joining the Pope Center in 2006, Shaw spent 22 years in ...

Most Popular

Sports

Madden 21’s Creepy Colin Kaepernick Coercion

Madden NFL 21, this year’s iteration of the only major football video game in the world, has an average Google user rating of 1.4 out of 5. On Metacritic, it’s .2 out of 10. At CGMagazine, Brock McLaughlin writes that “This game is much like 2020, a giant disaster” and urged those interested to buy it if ... Read More
Sports

Madden 21’s Creepy Colin Kaepernick Coercion

Madden NFL 21, this year’s iteration of the only major football video game in the world, has an average Google user rating of 1.4 out of 5. On Metacritic, it’s .2 out of 10. At CGMagazine, Brock McLaughlin writes that “This game is much like 2020, a giant disaster” and urged those interested to buy it if ... Read More
Politics & Policy

Pennsylvania Poll: Biden 49, Trump 44

In the final Pennsylvania poll of the 2020 presidential election conducted by Muhlenberg College on behalf of the newspaper Morning Call, Joe Biden leads Donald Trump 49 percent to 44 percent. The same pollster's previous survey, conducted between October 13 and October 20, showed Biden leading Trump 51 percent ... Read More
Politics & Policy

Pennsylvania Poll: Biden 49, Trump 44

In the final Pennsylvania poll of the 2020 presidential election conducted by Muhlenberg College on behalf of the newspaper Morning Call, Joe Biden leads Donald Trump 49 percent to 44 percent. The same pollster's previous survey, conducted between October 13 and October 20, showed Biden leading Trump 51 percent ... Read More
White House

Hell, Yes

Editor’s Note: If you would like to read more pros and cons on voting for President Trump, further essays on the subject, each from a different perspective, can be found here, here, here, here, and here. These articles, and the one below, reflect the views of the individual authors, not of the National ... Read More
White House

Hell, Yes

Editor’s Note: If you would like to read more pros and cons on voting for President Trump, further essays on the subject, each from a different perspective, can be found here, here, here, here, and here. These articles, and the one below, reflect the views of the individual authors, not of the National ... Read More

Another Pollster Sees a Trump Win

The Trafalgar Group’s Robert Cahaly is an outlier among pollsters in that he thinks President Trump will carry Michigan, Pennsylvania, or both, and hence be reelected with roughly 280 electoral votes. (I explained his thinking here.) Last week another pollster, Jim Lee of Susquehanna Polling and Research, ... Read More

Another Pollster Sees a Trump Win

The Trafalgar Group’s Robert Cahaly is an outlier among pollsters in that he thinks President Trump will carry Michigan, Pennsylvania, or both, and hence be reelected with roughly 280 electoral votes. (I explained his thinking here.) Last week another pollster, Jim Lee of Susquehanna Polling and Research, ... Read More
Politics & Policy

Some Answers for Nicholas Kristof

The New York Times columnist uses a series of “questions” — not all of them phrased in a way that would be acceptable on Jeopardy — to urge religious conservatives to adopt a more “nuanced” position on abortion. Q: “Why do so many [Christians] see fervent opposition to any abortion as a religious ... Read More
Politics & Policy

Some Answers for Nicholas Kristof

The New York Times columnist uses a series of “questions” — not all of them phrased in a way that would be acceptable on Jeopardy — to urge religious conservatives to adopt a more “nuanced” position on abortion. Q: “Why do so many [Christians] see fervent opposition to any abortion as a religious ... Read More