Randomized controlled trials (RCTs) are the gold standard in evaluating the effects of treatments. To be clinically meaningful, results must be relevant to specific patient populations in specific settings. ¹ Multiple factors determine the external validity (i.e., generalizability or applicability) of RCTs: patient characteristics, condition under investigation, drug regimens, costs, compliance, co-morbidities, and concomitant treatments. For practical reasons, trials cannot always take these factors fully into consideration (e.g., costs, poor compliance). Also, certain aspects of study design—eligibility criteria, study duration, mode of intervention, outcomes, adverse events assessment, or type of statistical analysis greatly influence the degree of generalizability, given an appropriate population of interest.

Clinicians and policymakers often distinguish between the efficacy and the effectiveness of an intervention. Efficacy trials (explanatory trials) determine whether an intervention produces the expected result under ideal circumstances. Effectiveness trials (pragmatic trials) measure the degree of beneficial effect under “real world” clinical settings. ² Hence, hypotheses and study designs of an effectiveness trial are formulated based on conditions of routine clinical practice and on outcomes essential for clinical decisions.

Efficacy and effectiveness exist on a continuum. Generalizability depends largely on the viewpoint of the observer and the condition under investigation. Baseline patient characteristics (e.g., sex, age, severity of the disease, racial groups) are primary factors in generalizability; thus, depending on the population of interest, generalizability of the same study can range from low to high. Geographic settings (urban versus rural) and health care systems can also be significant, factors, ¹ although geography may have less influence on generalizability of drug trials than trials of other interventions (e.g., screening programs, behavioral therapy).

Ensuring generalizability may compromise internal validity. Under everyday clinical settings, factors such as patient or doctor preferences, ^{3, 4} or patient-doctor relationships ^{5, 6} can influence response and compliance. Random allocation, allocation concealment, and blinding negate these factors, thereby increasing internal validity on the one hand and decreasing external validity on the other. Therefore, to some extent, the operational definition of “effectiveness trial” delineates the necessary trade-offs with internal validity. An ideal definition would balance this equilibrium at a point at which satisfactory internal validity accompanies a high degree of generalizability.

Systematic reviews, including meta-analyses, have become an increasingly important source of information for clinical practice. If well conducted, they synthesize large amounts of information and provide estimated effect sizes that have greater precision and generalizability than individual studies. ⁷ Distinguishing between efficacy and effectiveness contributes an important aspect to analyzing any body of clinical evidence. Furthermore, greater emphasis on effectiveness studies may lead to changes in presentation in systematic reviews and policy initiatives.

In this article we propose and test seven hallmarks of study design to create a tool that can help researchers and those producing systematic reviews to distinguish more readily and more consistently between efficacy and effectiveness studies.