Reporting biases is an umbrella term that covers a range of different types of biases. Previous groups have described seven types of reporting biases including publication bias, time-lag bias, multiple (duplicate) publication bias, location bias, citation bias, language bias and outcome reporting bias (Higgins & Green. 2011). Reporting biases has been described as the most significant form of scientific misconduct (Al-Marzouki et al. 2005) and unethical research practice (World Medical Association, 2008).
Various definitions for reporting bias have been proposed:
- The Dictionary of Epidemiology defines reporting bias as the “selective revelation or suppression of information (e.g., about past medical history, smoking, sexual experiences) or of study results.”
- The Cochrane Handbook states reporting bias arises “when the dissemination of research findings is influenced by the nature and direction of results.”
- The James Lind Library states “biased reporting of research occurs when the direction or statistical significance of results influence whether and how research is reported.”
The selective disclosure or withholding of information by parties involved in the topic selection, design, conduct, analysis, or dissemination of a study or research findings leads to the problem of reporting bias
Download the powerpoint: Reporting biases
McGauran and colleagues conducted a narrative review on reporting bias in the medical literature (2010). They identified 50 different types of pharmacological, surgical, diagnostic and preventative interventions where reporting bias was involved, including the withholding of data by manufacturers and regulators or the active attempt by manufacturers to suppress the publication of findings.
A systematic review by Jones and colleagues compared the outcomes of randomised controlled trials published in registered protocols with the peer-reviewed journal publication. One-third of studies had a discrepancy between the registered and reported outcomes. Thirteen per cent of trials introduced a new outcome in the published manuscript compared to the registered protocol.
In a cohort study of systematic reviews, Saini and colleagues found 86% of the included Cochrane reviews did not report the main adverse outcome (2014). Another study found considerable inconsistency in the reporting of adverse events (Scharf & Colevas, 2006). In 14 of the 22 included studies, the number of adverse events in the sponsor’s database differed from the published articles by 20% or more.
When more detailed information for interventions was analysed, (i.e. clinical study reports), over half (55%) of the previous risk of bias assessments were reclassified from ‘low’ risk of bias to ‘high’ (Jefferson et al. 2014).
Trials and systematic reviews are used by clinicians and policymakers to develop evidence-based guidelines and make decisions about the treatment or prevention of diseases. When the evidence base available to clinicians or policymakers is incomplete, healthcare decisions and recommendations are made on biased evidence and spurious claims.
Reporting biases can distort the results of trials (Al-Marzouki et al. 2005) and bias systematic reviews and meta-analyses (Furukawa et al. 2007), leading to poorly informed and potentially invalid treatment decisions.
Vioxx (Rofecoxib), a Cox-2 inhibitor manufactured by Merck for pain associated with osteoarthritis, provides an important example of under-reporting and mis-reporting of data which led to significant patient harm.
The first safety analysis of the largest study of Rofecoxib, found a 79% greater risk of death or serious cardiovascular event in one treatment group compared with the other (Weinblatt et al., 1999). This information was not disclosed, the trial continued and the cardiovascular risk associated with Rofecoxib was obscured in several ways (Krumholz et al., 2007).
There were also a number of significant undisclosed conflicts of interest among board members of Vioxx which were not a matter of public record at the time the trial was conducted or published (Krumholz et al., 2007). Merck now faces legal claims from nearly 30,000 people who experienced an adverse cardiovascular event while taking Rofecoxib.
If benefits are over-reported and harms are under-reported, clinicians, patients and the public will have a false sense of security about the safety of treatments. This results in unnecessary suffering and death (Cowley et al. 1993), perpetuates research waste and misguides future research (Glasziou & Chalmers, 2018).
Pre-study: Prospective registration of trials is significantly associated with the publication of trial results (adjusted OR 4.53, 95% CI 1.12-18.34, Chan et al., 2017). Thus, policies mandating the prospective registration for all clinical trials should be implemented by journal editors, regulators, research ethics committees, funders, and sponsors.
During the study: Open science practices such as making de-identified data, code for data management and statistical analyses publicly available through platforms such as the Open Science Framework would help identify and deter reporting biases.
Post-study: Reporting guidelines such as CONSORT can help guide researchers to improve the reporting of randomised trials (Moher et al., 2010).
Other checklists and tools have been developed to assess the risk of reporting biases in studies including, the Cochrane Risk of Bias Tool, GRADE and ORBIT-II (Page et al., 2018).