Substitution game bias

Substitution of the clinically important endpoint, or an exposure, with a surrogate marker for the disease.

Background

In his 1979 paper listing sources of bias in analytic research, David Sackett included substitution game and gave the definition as: “The substitution of a risk factor which has not been established as causal for its associated outcome.”

Sackett cites Yerushalmy: In: Controversy in Internal Medicine. lngelfinger et (II. (Eds). 1966. We have been unable to locate a copy of this work, but have identified other work by this author on this subject. In 1959, Yerushalmy and Palmer discussed the use of substitutes in medical research, as in a) substitutes for disease as outcome measures b) substitutes for factors suspected of causing disease.

The authors urged caution in using substitutes when attempting to prove causality, largely because the substitute might lack specificity and may not hold a direct relation with the disease.

Prentice (1989) recommended that appropriate surrogate markers of disease must correlate with the clinical outcome of interest, and also fully capture the effect of an intervention or exposure on that outcome. Whilst it can be easy to find surrogate markers that are correlated with the disease of interest, it is much more difficult for those markers to wholly capture effects of an intervention or exposure on the disease outcome.

Problems with the use of surrogate markers can include

  • Lack of precision in estimating the surrogate marker
  • Effect of the outcome back on the marker (a problem in retrospective studies: sometimes the marker is influenced by recent events)
  • The time frame needs to be appropriate; the marker must reflect exposure in the correct time frame relevant for the development of disease
  • Technical issues with measurement of biomarkers e.g. consistency over time, consistency between different laboratories.
  • Biological variation of biomarkers e.g. over the menstrual cycle among reproductive-age women needs to be understood and taken into account.

“Surrogate outcomes are often used in clinical trials as substitutes for final patient relevant outcomes. Advantages of surrogate outcomes over final outcomes are that they may occur faster or may be easier to assess, thereby shortening the duration, size, and cost of trials. A key rationale for the use of surrogate outcomes in trials is not only substitution but the prediction of treatment benefit in the absence of data on patient relevant outcomes.”

Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study. BMJ. 2013 Jan 29;346:f457

Example

An example is the use of CD4 cell count as a surrogate marker for symptomatic AIDS events. CD4 cell counts are well correlated with subsequent AIDS events, however, changes in CD4 counts do not fully predict changes in rates of subsequent AIDS events.

Impact

Ciana and colleagues analysed effect sizes published from trials using surrogate markers and compared them to effect sizes published from trials using patient-relevant outcomes.

“On average, trials using surrogate outcomes reported treatment effects that were 28% to 48% higher than those of trials using final patient relevant outcomes. Furthermore, we found that surrogate trials were twice as likely to report positive treatment effects as the final outcome trials. These findings were not explained by differences in risk of bias or other trial characteristics and are comparable
with the level of exaggeration of treatment effect attributed to inadequate allocation concealment.”

This analysis showed that on average, trials using surrogate primary outcomes reported larger treatment effects than a matched sample of trials using patient relevant primary outcomes.  These results show that findings from studies using surrogate markers must be interpreted with additional caution.

Preventive steps

Exposures and outcomes measured in health research must be considered carefully in order to avoid introducing bias and error.

Where a surrogate marker is used to investigate relationships with disease, the effect of the exposure or intervention on the surrogate marker should predict the effect on the clinical outcome (not merely be correlated). Equally, where proxy exposures are used to investigate associations between an exposure and disease, the relationship between the proxy and the exposure of interest must be well understood and be a predictive relationship.

Leave a Reply

Your email address will not be published. Required fields are marked *