Background

Data-dredging bias is a general category which includes a number of misuses of statistical inference (e.g. fishing, p-hacking), but each essentially involves probing the data in unplanned ways, finding and reporting an “attractive” result, without accurately conveying the course of analysis. For example, in nearly any analysis of data there are several “researcher degrees of freedom”— i.e., choices that must be made in the process of analysis. Ideally, these choices are guided by the principles of best practice and prespecified in a publicly available protocol. In contrast, p-hacking occurs when an initial analysis produces results which are close to being statistically significant, then, in absence of a study protocol, researchers can make analytic choices (e.g. how to handle outliers, whether to combine groups, including/excluding covariates) which will produce a statistically significant p-value.[1]

While many different choices might be defensible, a canonical case of p-hacking would involve trying out multiple different options and reporting the result which yields the lowest p-value (particularly when alternative choices generate values that do not yield a significant result). Such an analysis can often generate statistically significant results in absence of a true effect (i.e. ‘false positives’) and is thus unreliable.

Though some forms of data-dredging are lamentably common, it is important to note that often such problems arise from a lack of awareness rather than malfeasance.[2] As Gelman and Loken (2014) note, “it can seem entirely appropriate to look at the data and construct reasonable rules for data exclusion, coding, and analysis that can lead to statistical significance” (p. 461).[3] In such cases an unconscious tendency to interpret the results in a biased fashion can be guarded against by prespecifying the course of analysis.

Apart from p-hacking, other forms of data dredging include: assessing models with multiple combinations of variables and selectively reporting the “best” model (i.e., “fishing”);[4] making decisions about whether to collect new data on the basis of interim results; making post-hoc decisions about which statistical analyses to conduct; and generating a hypothesis to explain results which have already been obtained but presenting it as it were a hypothesis one had prior to collecting the data (i.e., HARKing (“hypothesizing after the results are known”)).[5] In general, these procedures are acceptable when transparently reported; however, when authors neglect to accurately report how the results were in fact generated, they are rightfully classified as data-dredging.

Example

Despite numerous published trials and meta-analyses which appear to support the use of progestogens to mitigate pregnancy loss, when trials were limited to those that have been preregistered the evidence does not support their use.[6] Of 93 randomized controlled trials, 22 were classified as unlikely to be p-hacked. Of these, only one produced a statistically significant result and a meta-analysis of the trials found evidence that progesterone was not effective (RR = 1.00, 95% CI 0.94-1.07). In contrast, most previous meta-analyses support the use of progestogens. It is unlikely that this difference is the result of publication bias since, given the ratio between non-significant to significant results in pre-registered trials, we should expect an enormous number of unpublished studies which found no statistically significant effect. Prior et. al. therefore suggest that the difference is likely due to the inclusion of studies susceptible to data-dredging in the previous meta-analyses.[6]

The work of Brian Wansink and the Cornell Food and Brand Lab presents a more extreme example, which has nevertheless become a paradigm case of data-dredging. In a November 2016 blog post Wansink described how, at his encouragement, a visiting scholar reanalyzed the data from a “failed” study a generated four articles supporting claims such as food tastes worse when buffet prices are low, and men eat more when dining with women in order to impress them. Investigations revealed that Wansink and his team strategized how to generate statistical analyses which would produce flashy results. Crucially, the process which the team engaged in to generate the results was not revealed in the published papers. Instead, the results were presented as if hypotheses were developed before the data were gathered. Further investigation uncovered almost a decade’s worth of emails in which Wansink and his team strategized ways to dredge through data to find results they felt would be easy to publish, including correspondence with the visiting scholar in which Wansink requests that she “squeeze some blood out of this rock”.[7]

Impact

Though there is no definitive account of the frequency of data-dredging or the severity of the bias it induces, evidence for its existence derives from the fact that there is an unusually large number of published studies with a p-value slightly less than p = .05.[8, 9, 10, 11] While some authors have concluded that p-hacking is unlikely to have had a significant effect on meta-analytic estimates,[12] this assumes that most meta-analyses will include a number of studies with large sample sizes. Authors using p-curve analyses have found that the distribution of p-values is consistent with most research investigating real effects; however, the data is also consistent with some forms of data-dredging.[13, 14] What is clear is that the bias induced by data-dredging will be most severe in cases where the effect size is small, the dependent measures are imprecise, research designs are flexible, and studies are conducted on small populations.[15]

Preventive steps

John, Loewenstein, & Prelec (2012) find that researchers were generally unaware that data-dredging would induce bias.[2] Researchers frequently endorsed flawed practices such as deciding whether to gather more data after inspecting interim results or whether to exclude outliers after assessing the impact of doing so. Accordingly, there is hope that better statistical education would be beneficial. Authors should prespecify rules for stopping data collection, how to analyse data (including how to handle outliers, any expected transformations of the variables, what covariates will be controlled for, etc.) and studies should be sufficiently powered to detect meaningful effects. Publication of studies should list all variables collected in the study, report all planned analyses (specifying primary and secondary outcomes), and include robustness analyses for methodological choices.[1] Optimally, these choices should be registered prior to beginning the study[15]. When methodological decisions were informed by the data collected, the results should be clearly identified as an exploratory analysis and replicated.

The p-curve analysis has been suggested as a formal procedure for correcting p-hacking;[16, 17] however, its validity has not yet been established empirically. Particularly in non-randomized studies, confounding can render the p-curve analysis unreliable.[14]

Ultimately, as Banks et. al. (2016) have argued, data-dredging is a problem of bad barrels rather than bad apples because research systems incentivize producing nice-looking results.[19] Thus, one effective means of intervention would be to change the standards for conceiving, conducting, and publishing scientific research.[20] For example, the acceptance of articles on the basis of their design rather than their results would significantly alleviate the pressure to dredge data for “attractive” results.[21] Such alterations would represent sweeping changes for most disciplines.