Reporting Biases

The selective disclosure or withholding of information by parties involved in the design, conduct, analysis, or dissemination of a study or research findings.

Background

Reporting biases is an umbrella term that covers a range of different types of biases. Previous groups have described seven types of reporting biases including publication bias, time-lag bias, multiple (duplicate) publication bias, location bias, citation bias, language bias and outcome reporting bias (Higgins & Green. 2011). Reporting biases has been described as the most significant form of scientific misconduct (Al-Marzouki et al. 2005) and unethical research practice (World Medical Association, 2008).

Various definitions for reporting bias have been proposed:

  • The Dictionary of Epidemiology defines reporting bias as the “selective revelation or suppression of information (e.g., about past medical history, smoking, sexual experiences) or of study results.”
  • The Cochrane Handbook states reporting bias arises “when the dissemination of research findings is influenced by the nature and direction of results.”
  • The James Lind Library states “biased reporting of research occurs when the direction or statistical significance of results influence whether and how research is reported.

The selective disclosure or withholding of information by parties involved in the topic selection, design, conduct, analysis, or dissemination of a study or research findings leads to the problem of reporting bias

Download the powerpoint: Reporting biases

 

Example

McGauran and colleagues conducted a narrative review on reporting bias in the medical literature (2010). They identified 50 different types of pharmacological, surgical, diagnostic and preventative interventions where reporting bias was involved, including the withholding of data by manufacturers and regulators or the active attempt by manufacturers to suppress the publication of findings.

A systematic review by Jones and colleagues compared the outcomes of randomised controlled trials published in registered protocols with the peer-reviewed journal publication. One-third of studies had a discrepancy between the registered and reported outcomes. Thirteen per cent of trials introduced a new outcome in the published manuscript compared to the registered protocol.

In a cohort study of systematic reviews, Saini and colleagues found 86% of the included Cochrane reviews did not report the main adverse outcome (2014). Another study found considerable inconsistency in the reporting of adverse events (Scharf & Colevas, 2006). In 14 of the 22 included studies, the number of adverse events in the sponsor’s database differed from the published articles by 20% or more.

When more detailed information for interventions was analysed, (i.e. clinical study reports), over half (55%) of the previous risk of bias assessments were reclassified from ‘low’ risk of bias to ‘high’ (Jefferson et al. 2014).

Impact

Trials and systematic reviews are used by clinicians and policymakers to develop evidence-based guidelines and make decisions about the treatment or prevention of diseases. When the evidence base available to clinicians or policymakers is incomplete, healthcare decisions and recommendations are made on biased evidence and spurious claims.

Reporting biases can distort the results of trials (Al-Marzouki et al. 2005) and bias systematic reviews and meta-analyses (Furukawa et al. 2007), leading to poorly informed and potentially invalid treatment decisions.

Vioxx (Rofecoxib), a Cox-2 inhibitor manufactured by Merck for pain associated with osteoarthritis, provides an important example of under-reporting and mis-reporting of data which led to significant patient harm.

The first safety analysis of the largest study of Rofecoxib, found a 79% greater risk of death or serious cardiovascular event in one treatment group compared with the other (Weinblatt et al., 1999). This information was not disclosed, the trial continued and the cardiovascular risk associated with Rofecoxib was obscured in several ways (Krumholz et al., 2007).

There were also a number of significant undisclosed conflicts of interest among board members of Vioxx which were not a matter of public record at the time the trial was conducted or published (Krumholz et al., 2007). Merck now faces legal claims from nearly 30,000 people who experienced an adverse cardiovascular event while taking Rofecoxib.

If benefits are over-reported and harms are under-reported, clinicians, patients and the public will have a false sense of security about the safety of treatments. This results in unnecessary suffering and death (Cowley et al. 1993), perpetuates research waste and misguides future research (Glasziou & Chalmers, 2018).

Preventive steps

Pre-study: Prospective registration of trials is significantly associated with the publication of trial results (adjusted OR 4.53, 95% CI 1.12-18.34, Chan et al., 2017). Thus, policies mandating the prospective registration for all clinical trials should be implemented by journal editors, regulators, research ethics committees, funders, and sponsors. 

During the study: Open science practices such as making de-identified data, code for data management and statistical analyses publicly available through platforms such as the Open Science Framework would help identify and deter reporting biases.

Post-study: Reporting guidelines such as CONSORT can help guide researchers to improve the reporting of randomised trials (Moher et al., 2010).

Other checklists and tools have been developed to assess the risk of reporting biases in studies including, the Cochrane Risk of Bias ToolGRADE and ORBIT-II (Page et al., 2018).

Sources

Definitions of some types of reporting biases. Cochrane Handbook for Systematic Reviews of Interventions, v5.1.0. 2011.

The effect of scientific misconduct on the results of clinical trials: a Delphi survey. Contemp Clin Trials 2005. 26: 331-337.

Declaration of Helsinki. Ethical Principles for Medical Research Involving Human Subjects. 2008.

Reporting bias in medical research – a narrative review. Trials 2010. 11:37.

Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med 2015; 13:282.

Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ 2014; 349.

Adverse event reporting in publications compared with sponsor database for cancer clinical trials. J Clin Oncol 2006. 24:24 pp 3933-8.

Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. JAMA 2007. 297:468-470.

The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction: an example of publication bias. Int J Cardiol 1993. 1;40(2): 161-6.

Research waste is still a scandal – an essay by Paul Glasziou and Iain Chalmers. The BMJ 2018; 363:k4645.

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review. BMJ Open 2018; 8:e019703.

Association of trial registration with reporting of primary outcomes in protocols and publications. JAMA 2017; 318:17, 1709-1711.

Promoting public access to clinical trial protocols: challenges and recommendations. Trials 2018; 19:116.

Conflict of interest disclosure in biomedical research: a review of current practices, biases, and the role of public registries in improving transparency. Research Integrity and Peer Review 2016; 1:1.

Interim non-endpoint safety analysis of VIGOR—unblinded minutes. November 18, 1999. Found in sNDA S-007: P088C: Appendix 3.9.2 at p.2939-2946 (Bates No. MRK-00420015464-MRK-00420015471.

What have we learnt from Viixx? BMJ, 2007. 334(7585): 120-123

CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. 2010. 340:c869


PubMed feed

View more →