The New Health Care: Peer Review: The Worst Way to Judge Research, Except for All the Others

The New Health Care: Peer Review: The Worst Way to Judge Research, Except for All the Others

Unesco estimates that women make up 29 percent of the worldwide science work force.

But there are also data to support the value of peer review. A 1994 study, published in Annals of Internal Medicine, reviewed the quality of papers submitted to the journal before and after the peer review and editorial system. Researchers used a tool that assessed the manuscript’s quality on 34 items, and their work showed that all but one got better. The biggest improvements were in the discussion of a study’s limitations, its generalizations, its use of confidence intervals and the tone of the conclusions. Probably none of these would have occurred without the nudge of peer review.

How then to improve the existing system?

For starters, more formal training might improve quality and speed. Given how hard it is to recruit good reviewers, journal editors could consider better incentives, such as paying reviewers for their time. The unpaid costs of peer review were estimated at 1.9 billion pounds (almost $3.5 billion) in 2008. Or journals could offer, without promise of acceptance, quicker turnaround for a reviewer’s future papers. Academia might offer more formal recognition for review work as well.

A number of journals have moved toward fully blinded reviews, in which reviewers don’t know the authors or institutions of papers they’re judging. This could eliminate some biases. It’s hard to do this, though, because papers often refer to prior work or to where the research occurred. It also doesn’t solve the relative lack of women in the editorial and review process in general.

One way to detect problems with research earlier would be to let researchers post manuscripts online before submission, for public judgment before formal peer review. This is already common in some sciences, such as physics. Medical journals would probably resist this, however, because it could reduce their ability to get press and attention once the research was fully published.

A significant improvement would require a change in attitude. Too often, we think that once a paper gets through peer review, it’s “truth.” We’d do better to accept that everything, even published research, needs to be reconsidered as new evidence comes to light, and subjected to more thorough post-publication review.

As an author of papers, and as a writer who comments on papers in the news media, I’ve seen how the peer review process can fail. But I’m also an editor at the journal JAMA Pediatrics. There, as at many journals, a paper’s first gatekeeper is an editor. Those getting past that hurdle are sent out to a few experts in the field who are asked to read and offer their views to the editor. This informs what might happen next: acceptance, rejection or a chance to respond to reviewer comments before a decision is made.

Each week we meet by teleconference to discuss papers we are considering for publication. We talk about the reviews, and ultimately decide what few studies make the cut. I’m always impressed by the quality of the discussion and the seriousness with which people take their charge. We also follow papers we turn down to see if we made mistakes in deciding to reject. We use that data to review and improve our process. I’m sure other journals do the same. And I’m sure we make our share of bad calls, as other journals do.

Peer review is still better than the alternatives. It might make more sense, though, to see it (and publication) as steps on the road to assurance, not a final stamp of approval.


Please enter your comment!
Please enter your name here