Background

Reporting biases is an umbrella term that covers a range of different types of biases. It is described as the most significant form of scientific misconduct (Al-Marzouki et al. 2005). Reporting biases have been recognised for hundreds of years, dating back to the 17th century (Dickersin & Chambers, 2010). Since then, various definitions of reporting biases have been proposed:

• The Dictionary of Epidemiology defines reporting bias as the “selective revelation or suppression of information (e.g., about past medical history, smoking, sexual experiences) or of study results.”
• The Cochrane Handbook states it arises “when the dissemination of research findings is influenced by the nature and direction of results.”
• The James Lind Library states “biased reporting of research occurs when the direction or statistical significance of results influence whether and how research is reported.”

Our definition of reporting biases is a distortion of presented information from research due to the selective disclosure or withholding of information by parties involved with regards to the topic selected for study and the design, conduct, analysis, or dissemination of study methods, findings or both. Researchers have previously described seven types of reporting biases, including publication bias, time-lag bias, multiple (duplicate) publication bias, location bias, citation bias, language bias and outcome reporting bias (Higgins & Green. 2011). Figure 1 illustrates where reporting biases can occur in the lifecycle of research and provides several examples of reporting biases.

Download the powerpoint: Reporting biases

Example

A narrative review conducted by McGauran and colleagues (2010) found reporting biases are a widespread phenomenon in the medical literature. They identified reporting biases in 50 types of pharmacological, surgical, diagnostic and preventative interventions which included the withholding of study data or the active attempt by manufacturers to suppress the publication of findings.

A systematic review by Jones and colleagues (2015) compared the outcomes of randomised controlled trials specified in registered protocols with those in subsequent peer-reviewed journal articles. There were discrepancies between prespecified and reported outcomes in a third of the studies. Thirteen per cent of trials introduced a new outcome in the published articles compared with those specified in the registered protocols.

In a cohort study of Cochrane systematic reviews, Saini and colleagues (2014) found 86% of reviews did not report data on the main harm outcome of interest.

Another cohort study found considerable inconsistency in the reporting of adverse events when comparing sponsors databases with study protocols (Scharf & Colevas, 2006). In 14 of the 22 included studies, the number of adverse events in the sponsor’s database differed from the published articles by 20% or more.

When more detailed information for interventions was analysed for oseltamivir trials, over half (55%) of the previous risk of bias assessments were reclassified from ‘low’ risk of bias to ‘high’ (Jefferson et al. 2014).

Impact

Trials and systematic reviews are used by clinicians and policymakers to develop evidence-based guidelines and make decisions about treatment or prevention of health problems. When the evidence base available to clinicians, policymakers or patients is incomplete or distorted, healthcare decisions and recommendations are made on biased evidence.

Vioxx (Rofecoxib), a Cox-2 inhibitor prescribed for osteoarthritis pain, provides an important example of under-reporting and misreporting of data which led to significant patient harm.

The first safety analysis of the largest study of Rofecoxib found a 79% greater risk of death or serious cardiovascular event in one treatment group compared with the other (Krumholz et al., 2007). This information was not disclosed by the manufacturer (Merck), and the trial continued. The cardiovascular risk associated with Rofecoxib was obscured in several ways.

A number of significant conflicts of interest among board members of Merck were undisclosed, and not made public while the trial was in progress or when it was published (Krumholz et al., 2007). Merck now faces legal claims from nearly 30,000 people who experienced adverse cardiovascular events while taking Rofecoxib.

If benefits are over-reported and harms are under-reported, clinicians, patients and the public will have a false sense of security about the safety of treatments. This results in unnecessary suffering and death (Cowley et al. 1993), perpetuates research waste and misguides future research (Glasziou & Chalmers, 2018).

Preventive steps

Transparency is the most important action to safeguard health research.

Pre-study: The results of prospectively registered trials are significantly more likely to be published than those of unregistered trials (adjusted OR 4.53, 95% CI 1.12-18.34; Chan et al., 2017). Prospective registration of all clinical trials should be required, and encouraged for other study designs, by journal editors, regulators, research ethics committees, funders, and sponsors. 

During the study: Open science practices, such as making de-identified data and analytical code publicly available through platforms like GitHub or the Open Science Framework aids reproducibility, prevents duplication, reduces waste, accelerates innovation, identifies errors and prevents reporting biases.

Post-study: Reporting guidelines such as CONSORT can help guide researchers to improve their reporting of randomised trials (Moher et al., 2010).

Other checklists and tools have been developed to assess the risk of reporting biases in studies, including the Cochrane Risk of Bias Tool, GRADE and ORBIT-II (Page et al., 2018).