Academics who have written papers usually get no credit for them unless they are published in “peer reviewed” journals. Supposedly, that filters out poor work.
But does it actually work that way? In today’s Martin Center article, Richard Phelps argues that the peer-review process is not reliable. The incentives for reviewers to do insightful work are almost nonexistent, and the opportunities for them to wreak havoc for ideological reasons are numerous.
Phelps begins with a personal example, a manuscript of his that was buffeted by reviewer comments about “tone” and other petty complaints.
“Few of my manuscript submissions to education journals,” Phelps continues, “have been reviewed substantively. For those familiar with Paul Graham’s hierarchy of disagreement, the reviews tend to lack a refutation of the central point, refutation in general, or counterargument. Instead, most consist of responding to (perceived) tone or being ad hominem (even anonymously) attacks.”
Top-notch scholars are usually too busy to bother with reviewing, so the task falls mainly to young ones, who increasingly use their reviews to do hatchet jobs.
The peer-review system has become very corrupt, Phelps argues: “University and faculty raters construct their metrics in part from peer review results. Gaming these metrics corrupts the process with fake co-authors or reviews, citation cartels, co-citation agreements between journals or universities, reviewer- or editor-coerced citations, or purchased citations or by-lines.”
A better system, he suggests, would be for scholars to publish first and then allow people to comment, and not anonymously.
Phelps concludes, “We may forget, however, that citation and publication numbers are not supposed to be ends in themselves but, rather, just proxies for adding useful and accurate information to society’s store of knowledge. Society might be better served by fewer scholarly journals publishing fewer articles that have been much more widely and thoroughly vetted.”