X

Create an account to continue reading.

Registered readers have access to our blogs and a limited number of magazine articles
For unlimited access to The Spectator, subscribe below

Registered readers have access to our blogs and a limited number of magazine articles

Sign in to continue

Already have an account?

What's my subscriber number?

Subscribe now from £1 a week

Online

Unlimited access to The Spectator including the full archive from 1828

Print

Weekly delivery of the magazine

App

Phone & tablet edition of the magazine

Spectator Club

Subscriber-only offers, events and discounts
 
View subscription offers

Already a subscriber?

or

Subscribe now for unlimited access

ALL FROM JUST £1 A WEEK

View subscription offers

Thank you for creating your account – To update your details click here to manage your account

Thank you for creating your account – To update your details click here to manage your account

Thank you for creating an account – Your subscriber number was not recognised though. To link your subscription visit the My Account page

Thank you for creating your account – To update your details click here to manage your account

X

Login

Don't have an account? Sign up
X

Subscription expired

Your subscription has expired. Please go to My Account to renew it or view subscription offers.

X

Forgot Password

Please check your email

If the email address you entered is associated with a web account on our system, you will receive an email from us with instructions for resetting your password.

If you don't receive this email, please check your junk mail folder.

X

It's time to subscribe.

You've read all your free Spectator magazine articles for this month.

Subscribe now for unlimited access – from just £1 a week

You've read all your free Spectator magazine articles for this month.

Subscribe now for unlimited access

Online

Unlimited access to The Spectator including the full archive from 1828

Print

Weekly delivery of the magazine

App

Phone & tablet edition of the magazine

Spectator Club

Subscriber-only offers, events and discounts
X

Sign up

What's my subscriber number? Already have an account?

Thank you for creating your account – To update your details click here to manage your account

Thank you for creating your account – To update your details click here to manage your account

Thank you for creating an account – Your subscriber number was not recognised though. To link your subscription visit the My Account page

Thank you for creating your account – To update your details click here to manage your account

X

Your subscriber number is the 8 digit number printed above your name on the address sheet sent with your magazine each week. If you receive it, you’ll also find your subscriber number at the top of our weekly highlights email.

Entering your subscriber number will enable full access to all magazine articles on the site.

If you cannot find your subscriber number then please contact us on customerhelp@subscriptions.spectator.co.uk or call 0330 333 0050. If you’ve only just subscribed, you may not yet have been issued with a subscriber number. In this case you can use the temporary web ID number, included in your email order confirmation.

You can create an account in the meantime and link your subscription at a later time. Simply visit the My Account page, enter your subscriber number in the relevant field and click 'submit changes'.

If you have any difficulties creating an account or logging in please take a look at our FAQs page.

Features

How many scientific papers just aren’t true?

Enough that basing government policy on ‘peer-reviewed studies’ isn’t all it’s cracked up to be

29 October 2016

9:00 AM

29 October 2016

9:00 AM

We’re continually assured that government policies are grounded in evidence, whether it’s an anti-bullying programme in Finland, an alcohol awareness initiative in Texas or climate change responses around the globe. Science itself, we’re told, is guiding our footsteps.

There’s just one problem: science is in deep trouble. Last year, Richard Horton, editor of The Lancet, admitted that “much of the scientific literature, perhaps half, may simply be untrue.” In his words, “science has taken a turn toward darkness.”

Medical research, psychology, and economics are all in the grip of a ‘reproducibility crisis.’ A pharmaceutical company attempting to confirm the findings of 53 landmark cancer studies was successful in only six instances, a failure rate of 89pc. In 2012, a psychology journal devoted an entire issue to reliability problems in that discipline, with one essay titled “Why science is not necessarily self-correcting.” Likewise, a 2015 report prepared for the Board of Governors of the US Federal Reserve concluded that “economics research is usually not replicable.” Its authors were able to verify the findings of only one third of 67 papers published in reputable economics journals. After enlisting the help of the original researchers, the success rate rose to a still dismal 49pc.

Government policies can’t be considered evidence-based if the evidence on which they depend hasn’t been independently verified, yet the vast majority of academic research is never put to this test. Instead, something called peer review takes place. When a research paper is submitted, journals invite a couple of people to evaluate it. Known as referees, these individuals recommend that the paper be published, modified, or rejected.

If one gets what one pays for, it’s worth observing that referees typically work for free. They lack both the time and the resources to perform anything other than a cursory overview. Nothing like an audit occurs. No one examines the raw data for accuracy or the computer code for errors. Peer review doesn’t guarantee that proper statistical analyses were employed, or that lab equipment was used properly.


Referees at the most prestigious of journals have given the green light to research that was later found to be wholly fraudulent. Conversely, they’ve scoffed at work that went on to win Nobel Prizes. Richard Smith, a former editor of the British Medical Journal, describes peer review as a roulette wheel, a lottery, and a black box. He points out that an extensive body of research finds scant evidence that this vetting process accomplishes much at all. On the other hand, a mountain of scholarship has identified profound deficiencies.

Peer review’s random and arbitrary nature was demonstrated as early as 1982. Twelve already published papers were assigned fictitious author and institution names before being resubmitted to the same journal 18-32 months later. The duplication was noticed in three instances, but the remaining nine papers underwent review by two referees each. Only one paper was deemed worthy of seeing the light of day the second time it was examined by the same journal that had already published it. Lack of originality wasn’t among the concerns raised by the second wave of referees.

Anyone can start a scholarly journal and define peer review however they wish. No minimum standards apply and no enforcement mechanisms ensure that a journal’s publicly described policies are actually followed. Some editors admit to writing up fake reviews under the cover of anonymity rather than going to the trouble of recruiting bona fide referees. In 2014, a news story reported that 120 papers containing computer-generated gibberish had nevertheless survived the peer review process of reputable publishers.

Politicians and journalists have long found it convenient to regard peer-reviewed research as de facto sound science. If that were the case, Nature would hardly have subtitled a February 2016 article: “Mistakes in peer-reviewed papers are easy to find but hard to fix.” Over a period of 18 months, a team of researchers attempted to correct dozens of substantial errors in nutrition and obesity research. Among these was the claim that the height change in a group of adults  averaged nearly three inches (7 cm) over eight weeks. The team reported that editors “seemed unprepared or ill-equipped to investigate, take action or even respond.” In Kafkaesque fashion, after months of effort culminated in acknowledgement of a gaffe, journals then demanded that the team pay $1,700 in one instance and $2,100 in another before a letter calling attention to other people’s mistakes could be published.

Which brings us back to the matter of public policy. We’ve long been assured that reports produced by the UN’s Intergovernmental Panel on Climate Change (IPCC) are authoritative because they rely entirely on peer-reviewed, scientific literature. A 2010 InterAcademy Council investigation found this claim to be false, but that’s another story. Even if all IPCC source material did meet this threshold, the fact that one out of an estimated 25,000 academic journals conducted an unspecified and unregulated peer review ritual is no warranty that a paper isn’t total nonsense.

If half of the scientific literature “may simply be untrue,” then half of the climate research cited by the IPCC may also be untrue. This appalling unreliability extends to work on dietary cholesterol, domestic violence, air pollution – in short, to all research currently being generated by the academy.

The US National Science Foundation recently reminded us that a scientific finding “cannot be regarded as an empirical fact” unless it has been “independently verified.” Peer review does not perform that function. Until governments begin authenticating research prior to using it as the foundation for new laws and huge expenditures, don’t fall for the claim that policy X is evidence-based.

Donna Laframboise is the author of Peer Review: Why Scepticism is Essential, a report published by the Global Warming Policy Foundation. This replaces an earlier version of this article that had been published in error. 

Subscribe to The Spectator today for a quality of argument not found in any other publication. Get more Spectator for less – just £12 for 12 issues.


Show comments
Close