Section 3 discussed the strengths and limitations of RCTs. A 2011 AHRQ methods report described a “best evidence” framework that outlined some important considerations when determining whether evidence from RCTs alone may meet the goals of an SR.³⁸ The report also noted that different topics may require different decisions regarding the inclusion of NRSIs.

We recommend that several factors be considered to help determine whether RCTs or NRSIs will likely form the primary evidence base or if RCTs and NRSIs may complement each other. These interrelated considerations include:

1. What are the decisional dilemmas and Key Question being addressed, and how will the end user(s) of the SR use the evidence to inform decision making?
2. Is it logical and feasible for RCTs to address the Key Question either solely or primarily?

   • Population: What are the condition and population of interest? Are they likely to be adequately represented in RCTs?
   • Interventions and comparators: Are they new or established? Do considerable variations exist in how they are implemented in clinical practice? Are the full ranges of interventions and comparators of interest likely to be adequately represented in RCTs?
   • Outcomes: What are the primary outcomes of interest, including benefits and harms? Are there outcomes of interest (e.g., long-term effectiveness, long-term harms) for which RCTs may not be suitable, available, or feasible? Particularly for harms, is a comparator group required to answer the Key Question?
   • Settings: What settings are of primary importance? Are they likely to be adequately represented in RCTs?
3. How serious is the risk of bias in NRSIs that address the Key Question?³⁸

   • Is causal inference needed?
   • What level of methodologic rigor would be required of NRSIs to allow meaningful conclusions, meet the needs of the end-user, and comply with contemporary standards for SRs?
4. To what extent do NRSIs and RCTs complement each other?

   • Is randomization required to answer the Key Question, particularly regarding benefits?
   • Taken together, would the body of evidence from RCTs cover diverse populations and/or implementations of the intervention?

The above is not an exhaustive list of considerations. Useful information on many of these considerations may be apparent only during Topic Refinement or later. The decisions are not always straightforward and generally depend on the topic and Key Questions. In general, we agree with a strategy that considers risk of bias and applicability and assesses overall strength of evidence based on the best set of studies (i.e., the “best evidence” approach).³⁸ Whatever decision is made regarding including NRSIs, it is crucial for reviewers to be transparent in reporting the decision and a justification for it in the SR protocol and in the description of methods in the final report. Reporting study findings and considering their limitations (regardless of study design) remain of primary importance.