Report by Russ Dobler on July 18, 2016 event.
Note: An earlier version of this article misidentified Ben Goldacre's COMPare group as CONSORT, which is an unrelated set of recommendations for reporting randomized trials. The original article also incorrectly stated that no randomized trials had been conducted on the drug digoxin until recently.
It’s humbling to talk in America,” British academic and bestselling author Ben Goldacre told the New York City Skeptics, “because nobody knows who the fuck I am in America.”
A packed house at Baruch College on this wet, hot, American summer evening argued otherwise. The physician and writer of the books Bad Science and Bad Pharma was on the continent and in town on July 18 to talk about a specific way in which medical science goes wrong, with bad drug trials.
Tens of thousands of randomized controlled trials (RCTs) are conducted every year, Goldacre said, to avoid the pitfalls of just using observational data to judge a treatment’s efficacy. The heart failure and atrial fibrillation drug digoxin, for example, was prescribed for years to lengthen the lives of cardiovascular patients, based largely on anecdotal reports that it worked. Later metanalyses of the existing research instead showed that digoxin, in reality, had no positive effect on mortality, although it may decrease the number of required hospitalizations.
But even the best intentions can yield bad results when trials are conducted improperly. Goldacre came armed with a laundry list of methods with which researchers can manipulate data, knowingly or unknowingly, to fool academia, the press or even themselves.
About 30% of drugs approved by the U.S. Food and Drug Administration between the years 2000 and 2010 were tested with RCTs, but only against placebos, to see if the treatments worked better than no treatment at all. “That sounds superficially like the right thing to do,” Goldacre said, but the goal should actually be to find out if a treatment works better, and with fewer side effects, than existing drugs.
Of course, there’s room for deception when testing in direct comparison with other drugs, too. Some experimenters might use an unusually high dose of an existing treatment to accentuate its side effects, or a lower dose than needed of the subject drug, to mask its own. On the other side, pumping up the administered amount of a new drug might make it seem more effective than it actually is.
Choosing suitable participants for a trial is important, too, but not always for the reasons researchers think. “All too often, trials are conducted on ideal, perfect patients,” Goldacre said, when real life populations are “messy.” What good is it to bar marijuana-smokers from schizophrenia drug tests when many schizophrenics use it to help cope with symptoms? Some trials of clinical depression drugs don’t even accept people who drink alcohol.
And once a researcher has decided on selection criteria, that person needs to make sure someone else does the selecting. This kind of blinding can prevent experimenters from (consciously or unconsciously) tipping the scales by putting certain people in certain groups. The developer of a drug may want to put an especially sick person into the control group, so as to not make his own drug look bad if the patient doesn’t improve, for instance. Analyses have shown that drugs can seem to have up to 40% greater effects than they actually do if this allocation process is not done properly.
The whole thing becomes even murkier when you’re only testing what Goldacre called “surrogate” outcomes. "In the real world, the outcomes we’re concerned with are pain, suffering and death,” Goldacre said. Why is it then the things most-often tested are only related indicators, like cholesterol level and blood pressure?
“People take a long time to die,” Goldacre said of the practically of directly testing something like mortality. But using surrogate outcomes can leave you still wondering. While diabetes drugs are very good at lowering a patient’s blood sugar, not a single one has been shown to reduce the chances of amputations, heart attack or stroke.
That’s probably still better than switching what outcomes you’re testing for after the trial has begun. “You can give yourself so many opportunities to cherry pick your results,” Goldacre said, which can even lead to thinking that a treatment might be better for one group of people especially. A particularly famous (and poignant) instance was when British medical journal The Lancet insisted the ISIS-2 trial include a subgroup analysis, after the fact, to see if aspirin was more or less likely to prevent heart attacks in certain cohorts. Turns out it’s not very effective for Libras or Geminis.
This kind of "outcome switching" is the particular focus of the Centre for Evidence-Based Medicine's Outcome Monitoring Project (COMPare), of which Goldacre is a part. Between October 2015 and January 2016, the group collectively assessed the findings of every trial published in the world's most prestigious medical journals, to see if their findings were properly reported.
“We had three meetings a week; they were two hours long [each],” Goldacre said. “They were fucking miserable.” The results were perhaps even more painful, as just over half of all pre-specified outcomes were reported, and an average of five new outcomes were added to each study.
Things didn’t get much better when these observations were brought to the attention of those major medical journals. While the BMJ published corrections to the cited studies on their website, The Lancet waited six months to address the issue in print, with responses from the original trialists.
“This is no way to deal with the problem,” Goldacre said.
The Journal of the American Medical Association wanted nothing to do with COMPare at all, and didn’t respond when COMPare’s letters were corrected for being too “vague” and “repetitive.” The New England Journal of Medicine went the extra mile and noted all the mistakes COMPare itself had made. There was no response from the journal when Goldacre and company pointed out, with examples, how the accusation was factually incorrect.
More than half of the 58 total correction letters sent were rejected by journal editors.
In addition to working with COMPare, Goldacre helped found the AllTrials initiative in 2013, an effort to make sure that all clinical trials are reported, as currently only about half are ever published.