The current system gives some scientific errors high prestige packaging. 

A few years ago, I wrote a research paper about peer-review – the process by which academic journals decide which articles should be published, amongst the thousands submitted to them every year. Typically, journals invite third parties to critique these articles prior to making their publication/rejection decisions.

As part of my research, I read 400 pages of written and oral testimony about peer review presented to a UK House of Commons committee back in 2011. Little of that commentary contained as much clear thinking as a recently-published, six-page paper titled In peer review we (don’t) trust: How peer review’s filtering poses a systemic risk to science.

Two American statisticians, Harry Crane and Ryan Martin, argue persuasively that peer review gives everyone a false sense of security. In their words:

A widely held belief among academics is that…every article eventually gets published somewhere. Moreover, most research results, whether right or wrong, are insignificant in the sense that they have a negligible impact, if any, on the trajectory of science. This raises the question: why should the scientific community expend so much time and effort vetting and filtering results that are highly likely to be published somewhere anyway and, even after they are published, are likely to be of little or no consequence to the course of science?

While peer review is labour intensive and time consuming, it provides no guarantee that a paper is correct. For example, on November 1st, Nature published an article about ocean heat and climate change that attracted widespread media attention. Nic Lewis, an independent UK researcher with a math and physics background, immediately suspected something was amiss.

He outlined his concerns on November 6th and November 7th. By November 9th, one of the article’s authors declared: “I am working with my co-authors to address two problems that came to our attention since publication.” (On November 17th, Lewis discussed further areas of concern.)

As Lewis has noted, this article was “peer reviewed and published in the world’s premier scientific journal.” Nevertheless, it appears far from watertight. So what’s the point of all that “rigamarole of submission, rejection, revision, repeat?” Why do we consider this a good use of expensively trained brainpower?

In modern day academia, say statisticians Crane and Martin, the prestige of the journal conducting the peer review is what matters, not the exercise itself. It’s all about snobbery now:

Though of little consequence to scientific progress, the journal outlet is of supreme importance to career advancement…Not because articles published in Nature or Science are ‘more correct’ but rather because results published in these journals are projected to be more important, and are thus more prestigious…From this point of view, the current peer review process is about as impure and unscientific as could be, the intellectual equivalent of the selfie-stick in today’s navel-gazing academic culture. [bold added]

Perversely, the institution of peer review makes matters worse by entrenching erroneous findings and amplifying their reach. The fact that a paper was subjected to this imperfect screening process before appearing in a high-impact journal encourages everyone to accept its conclusions. Science has spoken, therefore skepticism is unwarranted. In Crane and Martin’s words:

the perception that peer review provides an effective filter causes researchers to let down their guard, increasing the chance that errors propagate downstream…we recommend simply removing the filter, in order to force the scientific community first to acknowledge the reality that low-quality does get published and then to adapt to that reality.

Solid science is surely more likely to emerge from what Crane and Martin term a “culture of doubt” than from complacency born of “the illusion of quality control” that the peer review system currently provides.

.

If what you’ve just read is useful or helpful,
please consider supporting this blog

LINKS:

• hat tip to Judith Curry, who mentions Crane & Martin’s In peer review we (don’t) trust: How peer review’s filtering poses a systemic risk to science, and comments on the ocean heat controversy here
• the Nature article is here
• Harry Crane and Ryan Martin are the co-founders of Researchers.One, “a platform for peer review and scholarly publication and initiative for intellectual reform” launched in September 2018
• my 2016 paper, Peer Review: Why Skepticism is Essential

→ Receive posts via e-mail by signing up on the right side of this page, above – or by following this blog on Facebook and Twitter.
→ Download or e-mail a PDF of this post by clicking the Print button under Share This below – then select the blue arrow beside PDF at the bottom left.