Despite the enormous amount of new information that medical research generates every year, uncertainty plays a major role in health care decisionmaking. The challenging task for clinical and health policy decisionmakers is to balance considerations about evidence, values, preferences, and resources, all of which are often fraught with uncertainty and conflicting perspectives.¹

GRADE (Grading of Recommendations Assessment, Development and Evaluation) has evolved as a widely used approach to communicate certainties and uncertainties in systematic reviews to readers and other stakeholders.^2,3 GRADE uses information about risk of bias, imprecision, inconsistency, indirectness, and reporting bias to categorize the degree of uncertainty concerning the correctness of findings into four grades of quality of evidence.

The Evidence-based Practice Center (EPC) program of the U.S. Agency for Healthcare Research and Quality (AHRQ) has made small adaptations to the GRADE system to meet its specific needs.^4,5 Guidance for EPCs refers to quality of evidence as “strength of evidence”(SOE) and defines it as the degree of confidence that estimates are close to the true effect and the likelihood that findings will remain stable over time (i.e., the likelihood that future studies will not have an important impact on the estimate of an effect).⁴ In this paper, we refer generically to the GRADE approach but, as necessary for clarity, specify that specific points or findings refer to just the EPC approach. Table 1 summarizes the EPC definitions of the four levels of SOE.

Table 1

Definitions of grades of strength of evidence from the EPC program guidance.

Decisionmakers who rely on the GRADE approach assume that estimates of effect that are graded as high SOE are “close to the true effect” and, therefore, will remain stable as new evidence emerges. By contrast, decisionmakers can interpret effect estimates that are graded as low SOE as quite likely to change as new evidence accrues. In a recent international survey, we determined that producers and users of systematic reviews associated each grade of SOE with a distinct likelihood that estimates of effect will remain stable as new evidence emerges (see Table 1).⁶

To date, the predictive validity of the GRADE approach concerning the stability of effect estimates has not been tested. Predictive validity, in general terms, refers to the degree to which a score (such as the grades cited in Table 1) predicts an outcome on a criterion measure.⁷ For this analysis, predictive validity refers to the degree to which this approach, and specifically different SOE grades, reliably predicts the stability of an estimate of effect because it is close to the true effect.

A true effect can be viewed as the effect size that we would observe if a study had an infinitely large sample size (and thus no sampling error).⁸ Realistically, however, a true treatment effect can rarely be determined and used as a reference standard. For that reason, here we equate true effect with stability of effect as new studies emerge, a concept that can be measured. Given accurate predictive validity, a rating of “high SOE” would reliably predict that future studies will have a minor impact on the estimate of effect of a given outcome. Likewise, a rating of “low SOE” would reliably predict a high likelihood that future studies will have a substantial impact on the direction or magnitude of the estimate of effect of a given outcome.

The objective of our study was to determine the predictive validity of the EPC approach to GRADE based on a diverse sample of interventions. That is, we examined how reliably it can predict the likelihood that treatment effects remain stable.