Big Picture News, Informed Analysis

Canadian journalist Donna Laframboise. Former National Post & Toronto Star columnist, past vice president of the Canadian Civil Liberties Association.

Peer Review Is Bunk

A report I wrote for the Global Warming Policy Foundation was released today. It explains that peer-reviewed research is as likely to be wrong as right. Basing public policy on findings that haven’t yet been reproduced is nuts.

Cartoons by Josh

a marvelous cartoon by Josh graces the cover of my report

It’s time to slam on the brakes, folks. In recent decades, governments have justified all manner of guidelines, taxes, laws, and public awareness campaigns by claiming that a certain course of action is indicated by ‘science.’ We’re repeatedly told that ‘peer-reviewed’ science has determined X, and that society should therefore do Y.

But here’s the dirty little secret: the peer review process tells us almost nothing. It’s merely a sniff test. A couple of people briefly examine a research paper. Using entirely subjective criteria they decide that it kind of makes sense, that it must be right because it confirms their pre-existing beliefs, or that publishing it will gain the journal some wider media attention. Stripped of its mysticism, that’s all peer review is.

For decades, medical journals and others have tried to find evidence that peer review accomplishes something meaningful. It doesn’t. Peer review is a tool – a tool employed by the academic publishing industry to help it decide which papers to publish.

Just because an academic paper managed to get published in one of 25,000 peer-reviewed journals doesn’t mean its findings are sound. It’s a scandal that governments are confused about this. We don’t know if a paper’s results are accurate until we go to the trouble of reproducing it. Independent third parties must follow the same steps and achieve the same results. But almost no academic research is actually subjected to this test. When attempts are made to reproduce the findings of high profile research, the failure rate is often shocking.

Today, the Global Warming Policy Foundation has released a 40-page report authored by yours truly titled Peer Review: Why Skepticism is Essential.

Following a few months of intensive research, in which I read hundreds of pages of testimony to a UK parliamentary committee and examined peer review stretching back to its earliest days, my most charitable conclusion is this: Any particular piece of peer-reviewed research is just as likely to be wrong as right. One might as well flip a coin. We certainly shouldn’t be using it as a foundation for government action.

I’ve written a 900-word synopsis that begins:

We’ve all heard the buzzword. Whether it’s an anti-bullying program in Finland, an alcohol awareness initiative in Texas, or climate change responses around the globe, we’re continually assured that government policies are ‘evidence-based.’ Science itself guides our footsteps.

There’s just one problem: science is in deep trouble. Last year, Richard Horton, editor of The Lancet, admitted that “much of the scientific literature, perhaps half, may simply be untrue.” In his words, “science has taken a turn toward darkness.”

Medical research, psychology, and economics are all in the grip of a ‘reproducibility crisis.’ A pharmaceutical company attempting to confirm the findings of 53 landmark cancer studies was successful in only six instances, a failure rate of 89%. In 2012, a psychology journal devoted an entire issue to reliability problems in that discipline, with one essay titled “Why science is not necessarily self-correcting.” Likewise, a 2015 report prepared for the Board of Governors of the US Federal Reserve concluded that “economics research is usually not replicable.” Its authors were able to verify the findings of only one third of 67 papers published in reputable economics journals. After enlisting the help of the original researchers, the success rate rose to a still dismal 49%.

Government policies can’t be considered evidence-based if the evidence on which they depend hasn’t been independently verified, yet the vast majority of academic research is never put to this test. Instead, something called peer review takes place. When a research paper is submitted, journals invite a couple of people to evaluate it. Known as referees, these individuals recommend that the paper be published, modified, or rejected.

If one gets what one pays for, it’s worth observing that referees typically work for free. They lack both the time and the resources to perform anything other than a cursory overview. Nothing like an audit occurs. No one examines the raw data for accuracy or the computer code for errors. Peer review doesn’t guarantee that proper statistical analyses were employed, or that lab equipment was used properly.

Referees at the most prestigious journals have given the green light to research that was later found to be wholly fraudulent. Conversely, they’ve scoffed at work that went on to win Nobel Prizes. Richard Smith, a former editor of the British Medical Journal, describes peer review as a roulette wheel, a lottery, and a black box. CONTINUE READING

Download the 40-page report here.

Visit the Global Warming Policy Foundation website here.

Visit cartoons by Josh here.



Print Friendly, PDF & Email


This entry was posted on October 27, 2016 by in peer-review and tagged , .