6.1. Addressing the SR's Decisional Dilemma

When considering whether to include NRSIs in an SR, it is important to assess the extent to which the specific research questions addressed in relevant NRSIs align with the SR’s Key Questions. This assessment requires explicit consideration of whether the Key Questions, and hence the decisional dilemmas, concern efficacy, effectiveness, and/or harms.

As discussed in Section 1, RCTs, particularly explanatory RCTs, generally are designed to determine efficacy (i.e., the extent to which the intervention does more good than harm under the most ideal circumstances).¹⁰ NRSIs that are comparative generally focus on effectiveness (i.e., whether the intervention does more good than harm under the usual circumstances of routine practice).¹⁰ Comparative and single-group NRSIs also tend to provide evidence on the potential harms of an intervention that cannot be adequately gleaned from RCTs alone (see the EPC Methods Guide chapter Prioritization and Selection of Harms for Inclusion in Systematic Reviews³⁹). Because most EPC Program SRs of interventions aim to address both effectiveness and harms, it is not appropriate to routinely exclude all NRSIs even when numerous RCTs are available.

6.2. Applicability of NRSIs to the Specific PICO

The Cochrane Non-Randomized Studies of Interventions Methods Group offers guidance on assessing how well NRSIs address SR questions defined in terms of the populations, interventions, comparators, and outcomes (PICO).⁴ The assessment requires consideration of whether known available RCTs and NRSIs address a PICO-defined Key Question directly or indirectly. If RCTs address the Key Question directly and NRSIs address them only indirectly, it may be reasonable to exclude NRSIs from the SR. If both NRSIs and RCTs address the questions directly, it may be best to include NRSIs, especially if the RCT evidence is expected to contain gaps. However, NRSIs with high risk of bias may not be useful.

If the research question addressed in an NRSI (or RCT) deviates substantially from the PICO-defined SR question, it may be excluded. For example, in an SR of aspirin use for prevention of cardiovascular disease and colorectal cancer,⁴⁰ NRSIs (and RCTs) were excluded if they did not report on a comparison between aspirin and a non-aspirin control in a population eligible for primary prevention of cardiovascular disease.

Some outcomes may be particularly susceptible to measurement bias and confounding (see Section 7), so NRSIs that focus exclusively on such outcomes may be excluded if they report estimates of effectiveness or harms that are likely to be biased. For example, NRSIs, when compared with RCTs, have been shown to overestimate the benefits of treatments for pain.⁴¹ In an EPC SR of treatments for acute and chronic pain,⁴¹ reviewers focused on RCTs because of the susceptibility of NRSIs to confounding and bias for subjective outcomes, such as pain and function. However, the reviewers included large NRSIs for assessment of rare serious adverse events. For specific Key Questions where no RCTs were identified, the reviewers included cohort studies for evaluation of benefits.

6.3. Issues Related to Evolution of Evidence on a Topic

The decision regarding inclusion of NRSIs in an SR should consider the evolution of evidence in the topic area. Early in the evolution of evidence on a topic, when limited evidence from RCTs may be available, it is difficult to justify excluding NRSIs. In such a context, identification and synthesis of relevant NRSIs, perhaps with their limitations, could help articulate the need for RCTs.

As the evidence evolves, when RCT evidence accumulates, reviewers should consider whether and how NRSIs may have evolved to complement the evidence from the RCTs. The inclusion of NRSIs in SRs in this context could help (1) assess how the overall evidence applies to routine practice and specific subgroups of patients and (2) reveal the intervention’s long-term effectiveness and/or harms.

When the evidence matures to a stage where large seminal RCTs are available, NRSIs may be excluded if the outstanding evidence gaps are either not particularly important or if the evidence from NRSIs is unlikely to alter conclusions gleaned from RCTs.