Outcome Reporting Bias

The selective reporting of pre-specified outcomes in published clinical trials.

Background

Trials answer specific research questions about the benefits and harms of an intervention. The outcomes of interest should be specified by trialists before (a priori) the start. Outcomes should be selected based on their likelihood to impact on the patient’s overall health.

Selective reporting of pre-specified outcomes – depending on the nature and direction of the analysed results – occurs among a large proportion of published clinical trials, resulting in outcome reporting bias. Selective reporting of outcomes can potentially compromise the validity of a trial and any subsequent meta-analyses.

Selective reporting can occur through different ways:
1. omitting outcomes which are deemed to be unfavourable or statistically insignificant
2. adding new outcomes based on collected data to favour statistical significance
3. including only a subset of the analysed data in the published study
4. failing to report data that was analysed in the trial (such as adverse effects)
5. changing outcomes of interest (from primary outcomes to secondary outcomes if they do not yield significant results)

Outcome reporting bias can be difficult to detect. One way is to obtain the protocol of a clinical trial or trial registry (via databases such as Clinicaltrials.gov or the WHO clinical trials database) and compare the intended outcomes of interest to the analysed outcomes published in the final paper. If these differ then outcome reporting bias is present.

Example

A cohort study of trials and their protocols conducted in Denmark between 1994-1995 found that out of the 102 trials, 62% had changed, introduced or omitted at least one of their primary outcomes. A similar review of a cohort of Canadian studies found that 88% of the randomised trials failed to report at least one pre-specified outcome.

Chan AW et al. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171(7):735-40.

A re-analysis of 9 meta-analyses with previously unpublished US FDA trial data found that in two of them the effects were changed (Hart et al. 2012).

A systematic review in BMC Medicine included 27 analyses and found that in 31% (range 0 % to 100 %) of trials the primary outcomes were selectively reported.

A PLOS One systematic review of 137 RCTs from five high impact medical journals (The Lancet, British Medical Journal, New England Journal of Medicine, Annals of Internal Medicine and Journal of American Medical Association) found that 18% (25) had discrepancies relating to the primary outcome.

Impact

Studies that selectively omit or modify outcomes of interest have been shown to distort the overall treatment effect. Kirkham and colleagues conducted a sensitivity analysis of 81 reviews and found that in 23% the treatment effect was significantly reduced (i.e. by at least 20%). When 42 meta-analyses with only statistically significant outcomes were analysed, eight (19%) were no longer significant after adjusting for outcome reporting bias (eleven overestimated the treatment effect by at 20% or more).

The true scale of the harms of selective reporting is unknown although studies show it is a prevalent problem.

Selective reporting may cause misrepresentation of an intervention’s efficacy, skewing the public’s perception and collective scientific understanding of its benefits and harms. In a scientific community which has been shown to favour the publication of trials producing positive or ‘significant’ results, these trials are more likely to be published (Dwan et al. 2008).

Apart from misinforming patients and clinicians, if outcomes of interest are omitted based on statistical insignificance, withholding such results can result in other research groups conducting similar trials investigating for the same outcomes which could be wasteful research.

Preventive steps

While steps have been taken to try and mitigate the issue of outcome reporting bias, trials still change their pre-specified outcomes without justification.

In 1997, the FDA created the ClinicalTrials.gov website as a means for trial authors to register efficacy trials of serious and life-threatening illnesses. The International Committee of Medical Journal Editors called for the mandatory registration of trials in 2005 for trials to be published. The World Health Organization developed the International Clinical Trials Registry Platform (ICTRP) in 2006 for registration of trials involving patients as well as promoting clinical trial transparency.

In 2007, the US FDA Amendments Act in 2007 mandated trials done in the US had to be registered on ClinicalTrials.gov. EU Clinical Trial Regulation, requiring registration of trials on the EU Clinical Trials Register, will come into effect during 2019 instead of October 2018 as previously scheduled.

The COMPare Project is an initiative which targets five of the highest-impact factor journals in the medical field and audits outcome switching in randomised controlled trials published in these journals.

Clinical trial investigators must maintain a policy of transparency, and if they do change or omit planned outcomes, an adequate explanation should be provided for readers. Reviewers and journal editors should compare the final studies with their protocol or registry to assess for evidence of outcome reporting bias.

Sources

Chan A et al. Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles. JAMA. 2004;291(20):2457–2465.

Chan A et al. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171(7):735-40; doi: https://doi.org/10.1503/cmaj.1041086

Dwan K et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 2008;3:e3081.doi: https://doi.org/10.1371/journal.pone.0003081

Fleming PS et al. Outcome Discrepancies and Selective Reporting: Impacting the Leading Journals? PLoS One 2015;10(5):e0127495. doi:https://doi.org/10.1371/journal.pone.0127495

Hart B et al. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 2012;344:d7202; doi: https://doi.org/10.1136/bmj.d7202

Fleming PS et al. Outcome Discrepancies and Selective Reporting: Impacting the Leading Journals? PLoS One 2015;10(5):e0127495. doi:https://doi.org/10.1371/journal.pone.0127495

Ioannidis JPA et al. Outcome reporting bias in clinical trials: why monitoring matters BMJ 2017; 56 :j408; doi: https://doi.org/10.1136/bmj.j408

Jones CW et al. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Medicine 2014. doi: https://doi.org/10.1186/s12916-015-0520-3

Kirkham JJ et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews BMJ 2010; 340 doi: https://doi.org/10.1136/bmj.c365

Mathieu S et al. Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials. JAMA 2009;302(9):977-984. doi:10.1001/jama.2009.1242

Norris SL et al. Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 Aug. Available from: https://www.ncbi.nih.gov/books/NBK100617/?report=classic

Page MJ et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database of Systematic Reviews 2014, Issue 10. Art. No.: MR000035. DOI: 10.1002/14651858.MR000035.pub2.

The COMPare Trials Project. Goldacre B et al. www.COMPare-trials.org, 2016.


PubMed feed

View more →