Confounding

Although confounding is more of a concern for observational studies than RCTs, RCTs are not entirely free of these concerns. Confounding that is random (by chance) is expected to be equally distributed between arms by randomization, but RCTs may not always successfully randomize such potential confounders. Confounding by chance (that is confounding that is unknown, unmeasured, or poorly measured, but expected to be equally distributed) should occur with the same probability in RCTs and observational studies, because, it is, by definition, occurring by chance.⁷ Confounding by indication or contraindication, on the other hand, occurs when both treatment and outcome are influenced by a third factor, namely prognosis. Confounding by indication can occur when the expectation of prognosis influences the patient or provider's selection of treatment as well as the potential outcome. Trials, by virtue of concealed randomization, avoid this source of confounding, by “breaking the link between prognosis and prescription.”⁸, ^{p. e67} Observational studies of benefits that cannot address source of confounding by design (through, for instance, restriction to patients with the same prognosis) must account for it in analysis to the extent possible. Vandenbroucke notes that confounding by contraindication is not always a concern for observational studies of harms: harms are often unanticipated outcomes, so an expectation of prognosis for harms is unlikely to have influenced the selection of treatment.⁸, ^{p. e67} The extent to which confounding by indication may occur in observational studies of harms lies on a gradient, from the completely unanticipated (as with ACE inhibitors and angioneurotic edema) to the potentially likely (as with cardiac arrhythmias induced by anti-arrhythmic drugs).

Confounding by indication may also occur when the person allocating treatment is influenced by factors other than prognosis. Shrier et al. offer the example of the appearance of fatigue causing the physician to select one diabetes treatment over another.⁷ When confounding by indication can be controlled, for instance, by the inclusion of physician-rated appearance of patient fatigue as a covariate in modeling, the effect of the potential confounder can be accounted for. Each analysis need only account for those confounders that are expected to have an effect on the outcome and that have not already been accounted for by the inclusion of other closely correlated variables. Thus, the assessment of the potential for bias from confounding requires context-specific understanding of the relationship between treatment and outcomes and study-specific evaluation.^1,6,7

Selection Bias

Selection bias refers to the selection of the subset of those eligible, when that selection is conditioned upon variables that are the common effect of causes of the exposures and outcomes.^1,2 The specific risks of selection bias vary by type of observational study design. Differential loss to follow-up, for instance, is a concern for cohort studies but not for case-series or cross-sectional studies. On the other hand, a choice of control group that is intrinsically either more or less exposed than the source population of the cases, is a specific form of selection bias in case-control studies.

The concerns regarding attrition bias, however, are similar for RCTs and observational studies in theory: both types of studies may be weakened by high rates of overall or differential attrition. In practice, observational studies may have higher risks of attrition bias: they may have longer time horizons and fewer resources to follow up with participants

Lastly, RCTs often suffer less from inadvertent selection bias caused by the analysis. Usually, in RCTs, investigators will not adjust for other consequences of the treatment, nor will they adjust for consequences of the outcome. However, in large data-base analysis, that is sometimes done unwittingly, as with restriction of the analysis on folic acid supplementation on congenital malformation to live births.²

Performance Bias

The risk of performance bias may differ in practice between RCTs and observational studies. In a RCT the intervention is typically standardized so that study participants within groups are exposed (for the most part) to similar interventions or control conditions. Further, co-interventions can be standardized and/or monitored and fidelity to the intervention can be assessed and reported. For observational studies, this standardization may not be possible; therefore, researchers may not have control over how the intervention was administered or the level of exposure. As a result, observational studies may not be able to clearly define intervention states. Hernan and Taubman note that when the principle of consistency (a causal contrast between two or more well defined interventions) is not met (as with studies exploring whether obesity leads to increased mortality), other requirements of caual inference such as exchangeability and positivity cannot be met.⁹ This will also vary by specific observational study design features, particularly whether the intervention/exposure occurred prospectively vs. retrospectively with respect to the conduct of the study. Blinding of the providers and participants may also be of variable concern across different designs. For instance, in a retrospective study, the intervention or exposure would likely have been administered outside the context of a research study; therefore, blinding of the intervention/exposure may not be applicable. In a prospective study, by contrast, blinding of providers and participants is critical to limiting bias that arises as a result of knowing the study hypothesis and what the study participants are receiving. Blinding, or other measures that make assessments objective, is possible in assessing the exposure status of individuals in case-control studies.

Detection Bias

As with performance bias, detection bias may be more problematic in observational studies because outcome assessment may not be standardized as it is more typically with RCTs. For example, in RCTs the same outcome assessment tools are used often with protocols for their implementation and assessment of results. As above, this source of bias can also vary across observational studies, particularly for prospective vs. retrospective designs. In retrospective designs where the measurement of outcomes has occurred prior to the start of the study, the researchers have no control over how those assessments were made, including choice of measurement tools, whether tools were valid and reliable (or a process to ensure their validity/reliability was employed), and how results were interpreted. Blinding of outcome assessors serves to limit detection bias, but not all designs can employ a blinded approach to assessing outcomes. In some studies, researchers serve as outcome assessors and can be blinded; in other studies, participants provide self-reported outcome data and cannot be blinded. As above, blinding of outcome assessors may be of variable concern across different observational study designs.

Information bias is related to detection bias and arises from how measurement and assessment of exposure and outcomes are made. In theory, all designs run the risk of bias from the use of poorly validated measures.

Reporting Bias

Reporting bias is a concern across all research regardless of design: authors are more likely to publish studies and selectively report outcomes that show statistical significance. However, with the recent emphasis on standardized reporting of RCTs and prospective trial registration, reporting bias may become less problematic with RCTs or at least more easily detected as a problem in specific RCTs. While no empirical evidence exists that reporting bias is a greater concern for observational studies, fewer standards exist for observational studies, making reporting bias a concern of equal or greater magnitude than for RCTs.

References

1.

Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. 3rd ed. Philadelphia, PA: Lippincott, Williams, & Wilkins; 2008.

2.

Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004 Sep;15(5):615–25. [PubMed: 15308962]

3.

Viswanathan M, Berkman ND. Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol. 2012 Feb;65(2):163–78. [PubMed: 21959223]

4.

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.0.2. London: The Cochrane Collaboration; 2009. [June 9, 2011]. www​.cochrane-handbook.org.

5.

Juni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001 Jul 7;323(7303):42–6. [PMC free article: PMC1120670] [PubMed: 11440947]

6.

Greenland S, Morgenstern H. Confounding in health research. Annu Rev Public Health. 2001;22:189–212. [PubMed: 11274518]

7.

Shrier I, Boivin JF, Steele RJ, et al. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles. Am J Epidemiol. 2007 Nov 15;166(10):1203–9. [PubMed: 17712019]

8.

Vandenbroucke JP. Observational research, randomised trials, and two views of medical science. PLoS Med. 2008 Mar 11;5(3):e67. [PMC free article: PMC2265762] [PubMed: 18336067]

9.

Hernán MA, Taubman SL. Does obesity shorten life? The importance of well-defined interventions to answer causal questions. Int J Obes (Lond). 2008 Aug;32 Suppl 3:S8–14. [PubMed: 18695657]