Background
Trials answer specific research questions about the benefits and harms of an intervention. The outcomes of interest should be specified by trialists before (a priori) the start. Outcomes should be selected based on their likelihood to impact on the patient’s overall health.
Selective reporting of pre-specified outcomes – depending on the nature and direction of the analysed results – occurs among a large proportion of published clinical trials, resulting in outcome reporting bias. Selective reporting of outcomes can potentially compromise the validity of a trial and any subsequent meta-analyses.
Selective reporting can occur through different ways:
1. omitting outcomes which are deemed to be unfavourable or statistically insignificant
2. adding new outcomes based on collected data to favour statistical significance
3. including only a subset of the analysed data in the published study
4. failing to report data that was analysed in the trial (such as adverse effects)
5. changing outcomes of interest (from primary outcomes to secondary outcomes if they do not yield significant results)
Outcome reporting bias can be difficult to detect. One way is to obtain the protocol of a clinical trial or trial registry (via databases such as Clinicaltrials.gov or the WHO clinical trials database) and compare the intended outcomes of interest to the analysed outcomes published in the final paper. If these differ then outcome reporting bias is present.
Example
A cohort study of trials and their protocols conducted in Denmark between 1994-1995 found that out of the 102 trials, 62% had changed, introduced or omitted at least one of their primary outcomes. A similar review of a cohort of Canadian studies found that 88% of the randomised trials failed to report at least one pre-specified outcome.
Chan AW et al. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171(7):735-40.
A re-analysis of 9 meta-analyses with previously unpublished US FDA trial data found that in two of them the effects were changed (Hart et al. 2012).
A systematic review in BMC Medicine included 27 analyses and found that in 31% (range 0 % to 100 %) of trials the primary outcomes were selectively reported.
A PLOS One systematic review of 137 RCTs from five high impact medical journals (The Lancet, British Medical Journal, New England Journal of Medicine, Annals of Internal Medicine and Journal of American Medical Association) found that 18% (25) had discrepancies relating to the primary outcome.
Impact
Studies that selectively omit or modify outcomes of interest have been shown to distort the overall treatment effect. Kirkham and colleagues conducted a sensitivity analysis of 81 reviews and found that in 23% the treatment effect was significantly reduced (i.e. by at least 20%). When 42 meta-analyses with only statistically significant outcomes were analysed, eight (19%) were no longer significant after adjusting for outcome reporting bias (eleven overestimated the treatment effect by at 20% or more).
The true scale of the harms of selective reporting is unknown although studies show it is a prevalent problem.
Selective reporting may cause misrepresentation of an intervention’s efficacy, skewing the public’s perception and collective scientific understanding of its benefits and harms. In a scientific community which has been shown to favour the publication of trials producing positive or ‘significant’ results, these trials are more likely to be published (Dwan et al. 2008).
Apart from misinforming patients and clinicians, if outcomes of interest are omitted based on statistical insignificance, withholding such results can result in other research groups conducting similar trials investigating for the same outcomes which could be wasteful research.
Preventive steps
While steps have been taken to try and mitigate the issue of outcome reporting bias, trials still change their pre-specified outcomes without justification.
In 1997, the FDA created the ClinicalTrials.gov website as a means for trial authors to register efficacy trials of serious and life-threatening illnesses. The International Committee of Medical Journal Editors called for the mandatory registration of trials in 2005 for trials to be published. The World Health Organization developed the International Clinical Trials Registry Platform (ICTRP) in 2006 for registration of trials involving patients as well as promoting clinical trial transparency.
In 2007, the US FDA Amendments Act in 2007 mandated trials done in the US had to be registered on ClinicalTrials.gov. EU Clinical Trial Regulation, requiring registration of trials on the EU Clinical Trials Register, will come into effect during 2019 instead of October 2018 as previously scheduled.
The COMPare Project is an initiative which targets five of the highest-impact factor journals in the medical field and audits outcome switching in randomised controlled trials published in these journals.
Clinical trial investigators must maintain a policy of transparency, and if they do change or omit planned outcomes, an adequate explanation should be provided for readers. Reviewers and journal editors should compare the final studies with their protocol or registry to assess for evidence of outcome reporting bias.
