Medical journal editor admits science “has taken a turn towards darkness”

COMMENT OF DR. RICHARD HORTON, EDITOR-IN-CHIEF OF THE LANCET

Dr. Frankenstein

The following commentary was published in Britain’s oldest and most prestigious medical journal, The Lancet, in April, 2015.


Offline: What is medicine’s 5 sigma?

“A lot of what is published is incorrect.” I’m not allowed to say who made this remark because we were asked to observe Chatham House rules. We were also asked not to take photographs of slides. Those who worked for government agencies pleaded that their comments especially remain unquoted, since the forthcoming UK election meant they were living in “purdah”—a chilling state where severe restrictions on freedom of speech are placed on anyone on the government’s payroll. Why the paranoid concern for secrecy and non-attribution? Because this symposium—on the reproducibility and reliability of biomedical research, held at the Wellcome Trust in London last week—touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations.

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices. The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fi t their preferred theory of the world. Or they retrofit hypotheses to fit their data. Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of “significance” pollutes the literature with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.

Can bad scientific practices be fixed? Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative. Would a Hippocratic Oath for science help? Certainly don’t add more layers of research red tape. Instead of changing incentives, perhaps one could remove incentives altogether. Or insist on replicability statements in grant applications and research papers. Or emphasise collaboration, not competition. Or insist on preregistration of protocols. Or reward better pre and post publication peer review. Or improve research training and mentorship. Or implement the recommendations from our Series on increasing research value, published last year. One of the most convincing proposals came from outside the biomedical community. Tony Weidberg is a Professor of Particle Physics at Oxford. Following several high-profile errors, the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication. By filtering results through independent working groups, physicists are encouraged to criticise. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. Weidberg worried we set the bar for results in biomedicine far too low. In particle physics, significance is set at 5 sigma—a p value of 3 × 10–7 or 1 in 3·5 million (if the result is not true, this is the probability that the data would have been as extreme as they are). The conclusion of the symposium was that something must be done. Indeed, all seemed to agree that it was within our power to do that something. But as to precisely what to do or how to do it, there were no firm answers. Those who have the power to act seem to think somebody else should act first. And every positive action (eg, funding well-powered replications) has a counterargument (science will become less creative). The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.

Richard Horton
richard.horton@lancet.com

The Lancet, Vol 385, p 1380, April 11, 2015

Related

Why most published research findings are false

New England Journal of Medicine editor: “No longer possible to believe much of clinical research published”

Harvard Report: Institutional Corruption of Pharmaceuticals and the Myth of Safe and Effective Drugs

 

How Bad Medical Research is Destroying Our Health & Faith in Science

Our trust in science is waning. Do you believe what you hear from scientists? Like many Americans, you may be losing trust in what science is telling you.

A Huffington Post December 2013 poll found that only 36 percent of us have a “lot” of trust in the reliability of scientific information. 78 percent think that scientific studies are often or sometimes tainted by political ideology. And 82 percent believe scientific findings are often or sometimes influenced by the companies sponsoring the research. This is a disquieting trend.

This lack of confidence is especially strong regarding the nutritional and medical professions. One day it’s bad to eat eggs, the next day it’s good. A recent survey conducted by the International Food Information Council (IFIC) underscores how confused we really are:

Seventy-eight percent of the 1,002 respondents indicated they have encountered conflicting information about what constitutes a healthy food. Fifty-six percent felt that the confusion made them doubt the choices they were making food wise. Liz Sanders, director of research and partnerships at the foundation and a co-author of the survey said in a CNN report that:

“Americans rely on many different sources for their information when it comes to what foods to eat and what foods to avoid. Not all of these sources are really highly trusted, and it is likely that these sources share inconsistent information.”

How often have you found yourself or heard a family member dismissing a new nutritional study by saying “Forget it, these reports are too confusing and always changing. I’m going to eat and do what I like.” As it turns out, you and your family member share your confusion with a majority of Americans.

Why Are We Losing Trust In Science?

Setting aside the issue of religious fundamentalism, which can account for some of the mistrust of science, one possibility is the growing unreliability of many medical and nutritional practices and research. In the past few decades, there have been many medical and nutritional ideas that were later proven to be incorrect and even harmful.

Here’s a list of just a few confusing medical and nutritional practices prevalent in the past few decades that may have had a negative impact on your health:

  • Stents – stents cost about $30k a piece and are performed on over half a million Americans each year but there is little data indicating they prevent stroke.
  • High Carbohydrate Diets – touted as the cure for heart disease for over 40 years, they have led to an obesity and diabetes epidemic with no clear support for their efficacy.
  • Arthroscopic Knee Surgery – up to one million surgeries have been performed each year but a systematic review in the British Medical Journal found that for patients with meniscus tears it was no better than exercise therapy.
  • Vitamin Elong thought to prevent cancer and heart disease. Numerous international studies found no benefit for protection against heart disease, stroke or cancer.
  • Proton Pumps – the treatment of GERD costs Americans a billion dollars a year. Long term use (greater than 14 days) can lead to serious side-effects, such as  such as clostridium difficile infection, pneumonia, and malabsorption of calcium and magnesium leading to bone fractures and cardiac abnormalities.

As I’ve written elsewhere and as you can see from the above list, the health of millions of people is routinely compromised by flawed medical research and practices.

Medical journalist Richard Harris, makes some damaging revelations in Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope and Wastes Billions:

1 – the state of biomedical research

…medical researchers made much more progress between 1950 and 1980 than they did in the following three decades. Consider the development of blood-pressure drugs, chemotherapy, organ transplants, and other transformative technologies. Those all appeared in the decades before 1980.

2 – the development of new pharmaceutical drugs

The rate of drug approval has been falling since the 1950s…if you extrapolate the trend, starting in 1950, you’ll find that drug development comes to a halt in 2040.

In 2005 Dr. John P.A. Ioannidis, currently a professor in disease prevention at Stanford University, published the most widely accessed article in the history of the Public Library of Science (PLoS) entitled Why Most Published Research Findings Are False. It stated:

“There is increasing concern that most current published research findings are false.”

And that “…in modern research, false findings may be the majority or even the vast majority of published research claims.”

Ioannidis’ research model indicated that up to 80 percent of non-randomized research studies (the most common kind of study) are wrong, along with twenty-five percent of randomized trials (the supposed gold standard of research). Incredulously, these studies are published in top peer reviewed medical journals.

Much of what our physicians prescribe to us is wrong. Our doctors use research to inform their medical decisions – decisions about what drug to prescribe, which surgery to elect, and the best health strategy to adopt. They are often making crucial treatment decisions for depression, Alzheimer’s, type 2 Diabetes, cancer, obesity, etc. based on bad, incomplete or hidden medical research.

The Evolution of Bad Medical Science

Why is this happening? Why is so much of biomedical research so poorly done?

As it stands now, researchers are typically rewarded (tenure, grants, better jobs, etc.) for publishing a quantity of publications in prestigious journals. They do this by

  • Running small and statistically weak studies (they are easy to do) that produce only positive results since journals tend to not publish negative findings.
  • Ignoring negative findings.
  • Publishing only new and exciting findings that journals are looking for.
  • Never checking old findings for accuracy and replicability.
  • Changing methodologies in mid-stream to assure positive results.

The result is a proliferation of false and misleading findings that confuse scientists and the rest of us, waste billions of dollars and undermine the scientific endeavour and our health.

To tackle why this is happening, two scientists, Smaldino and McElreath, created a computational model based on evolutionary theory (natural selection) to explain the rising tide of weak biomedical research. They did this by simulating virtual research labs competing under conditions that exist today. The labs that put more effort into their research received fewer publications. The labs that published more received more grants, students and prestige.

The Results

Over numerous simulations and generations, the labs that were most successful (most publications, grants, etc.) passed their approach on to the next generation and proliferated. The labs that did not publish large amounts of research did not get the grants and prestige and did proliferate as readily.

As occurs in natural selection, the most successful labs recreated themselves with greater frequency.

Ed Yong of the Atlantic Monthly summarized the results: “Over time, and across many simulations, the virtual labs inexorably slid towards less effort, poorer methods, and almost entirely unreliable results.”

Conclusion

With the emphasis on quantity of publication, the forces of natural selection will continue to select for poor research, false findings, muddled methodology and accelerating confusion in the world of health and bio medical research. Replicability of findings will continue to falter as poor research studies abound.

The losers in all of this are those of us trying to make sense of what science is telling us so we can live better, more wholesome, and healthier lives.

Related CE Article:

Peer-Reviewed Science Losing Credibility As Large Amounts of Research Shown To Be False