Peter Berkowitz of the Hoover Institution has good piece in today’s WSJ on the flawed peer-review process in scinece. Subscriber only, but here’s the key excerpt:
Academic journals typically adopt a double blind system, concealing the names of both authors and reviewers. But any competent scholar can determine an article’s approach or analytical framework within the first few paragraphs. Scholars are likely to have colleagues and graduate students they support and whose careers they wish to advance. A few may even have colleagues whose careers, along with those of their graduate students, they would like to tarnish or destroy. There is no check to prevent them from benefiting their friends by providing preferential treatment for their orientation and similarly punishing their enemies.
That’s because the peer review process violates a fundamental principle of fairness. We don’t allow judges to be parties to a controversy they are adjudicating, and don’t permit athletes to umpire games in which they are playing. In both cases the concern is that their interest in the outcome will bias their judgment and corrupt their integrity. So why should we expect scholars, especially operating under the cloak of anonymity, to fairly and honorably evaluate the work of allies and rivals?
Some university presses exacerbate the problem. Harvard University Press tells a reviewer the name of a book manuscript’s author but withholds the reviewer’s identity from the author. It would be hard to design a system that provided reviewers more opportunity to reward friends and punish enemies.
Harvard Press assumes that its editors will detect and avoid conflicts of interest. But if reviewers are in the same scholarly field as, or in a field related to that of, the author — and why would they be asked for an evaluation if they weren’t? — then the reviewer will always have a conflict of interest.
Then there is the abuse of confidentiality and the overreliance on arguments from authority in hiring, promotion and tenure decisions. Owing to the premium the academy places on specialization, most university departments today contain several fields, and within them several subfields. Thus departmental colleagues are regularly asked to evaluate scholarly work in which they have little more expertise than the man or woman on the street.
Often unable to form independent professional judgments — but unwilling to recuse themselves from important personnel decisions — faculty members routinely rely on confidential letters of evaluation from scholars at other universities. Once again, these letters are written — and solicited — by scholars who are irreducibly interested parties.
There are no easy fixes to this state of affairs. Worse, our universities don’t recognize they have a problem. Instead, professors and university administrators are inclined to indignantly dismiss concerns about the curriculum, peer review, and hiring, promotion and tenure decisions as cynically calling into question their good character. But these concerns are actually rooted in the democratic conviction that professors and university administrators are not cut from finer cloth than their fellow citizens.