Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update

Structured Abstract

Contents

• Preface
• Acknowledgments
• Key Considerations
• 1. Introduction
  • 1.1. Rationale for Updated Guidance
  • 1.2. Considerations When Deciding Whether To Include NRSIs
  • 1.3. Scope of This Guidance
• 2. Workgroup Methods
  • 2.1. Composition of the Workgroup
  • 2.2. Obtaining Input From the Broader EPC Program
  • 2.3. Generating Consensus
  • 2.4. Writing This Report
• 3. Strengths and Limitations of RCTs
• 4. Types of NRSIs and Study Design Features
  • 4.1. Types of NRSIs
  • 4.2. Caution Regarding NRSI Designs
• 5. Key Considerations for Including or Excluding NRSIs
• 6. Applicability of NRSIs to the Key Questions
  • 6.1. Addressing the SR's Decisional Dilemma
  • 6.2. Applicability of NRSIs to the Specific PICO
  • 6.3. Issues Related to Evolution of Evidence on a Topic
• 7. Threats to Internal Validity of NRSIs
  • 7.1. Selection Bias
  • 7.2. Confounding
  • 7.3. Misclassification of Interventions
  • 7.4. Other Sources of Bias
• 8. Various Data Sources for NRSIs
• 9. Advanced Analytic Methods for NRSIs
  • 9.1. Advanced Analytic Approaches: Trial Emulation Efforts
  • 9.2. Advanced Analytic Approaches: Causal Inference Methods
  • 9.3. Summary of Considerations Regarding Advanced Methods
• 10. Incorporating NRSIs in Systematic Reviews
  • 10.1. Planning for the Inclusion of NRSIs
  • 10.2. Developing Searches for NRSIs
  • 10.3. Assessing the Risk of Bias in NRSIs
  • 10.4. Interpreting Results From NRSIs
  • 10.5. Incorporating Data From NRSIs Into Meta-Analyses
  • 10.6. Grading the Strength of the Body of Evidence That Includes NRSIs
  • 10.7. Reporting NRSI Evidence
• 11. Conclusion
  • 11.1. Summary of Guidance
  • 11.2. Context for Guidance and Challenges Encountered
• References
• Appendix A. Hedges

Suggested citation:

Saldanha IJ, Skelly AC, Vander Ley K, Wang Z, Berliner E, Bass EB, Devine B, Hammarlund N, Adam GP, Duan-Porter D, Bañez LL, Jain A, Norris SL, Wilt TJ, Leas B, Siddique SM, Fiordalisi CV, Patino-Sutton C, Viswanathan M. Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update. Methods Guide for Comparative Effectiveness Reviews. (Prepared by the Scientific Resource Center under Contract No. 290-2017-00003-C.) AHRQ Publication No. 22-EHC033. Rockville, MD: Agency for Healthcare Research and Quality; September 2022. DOI: https://doi.org/10.23970/AHRQEPCMETHODSGUIDENRSI. Posted final reports are located on the Effective Health Care Program search page.

This work was funded by the Agency for Healthcare Research and Quality through the following contracts: Scientific Resource Center (290-2017-00003-C), Brown University Evidence-based Practice Center (EPC) (75Q80120D00001), ECRI Institute–Penn Medicine EPC (75Q80120D00002), Pacific Northwest EPC (75Q80120D00006), Johns Hopkins University EPC (75Q80120D00003), Mayo Clinic EPC (75Q80120D00005), Minnesota EPC (75Q80120D00008), RTI–University of North Carolina EPC (75Q80120D00007), and University of Southern California EPC (75Q80120D00009). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.

The information in this report is intended to help healthcare decision makers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of healthcare services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report is made available to the public under the terms of a licensing agreement between the author and the Agency for Healthcare Research and Quality. Most AHRQ documents are publicly available to use for noncommercial purposes (research, clinical or patient education, quality improvement projects) in the United States, and do not need specific permission to be reprinted and used unless they contain material that is copyrighted by others. Specific written permission is needed for commercial use (reprinting for sale, incorporation into software, incorporation into for-profit training courses) or for use outside of the U.S. If organizational policies require permission to adapt or use these materials, AHRQ will provide such permission in writing.

AHRQ or U.S. Department of Health and Human Services endorsement of any derivative products that may be developed from this report, such as clinical practice guidelines, other quality enhancement tools, or reimbursement or coverage policies, may not be stated or implied. A representative from AHRQ served as a Contracting Officer’s Representative and reviewed the contract deliverables for adherence to contract requirements and quality.

AHRQ appreciates appropriate acknowledgment and citation of its work. Suggested language for acknowledgment: This work was based on a Methods Guide for Comparative Effectiveness Reviews paper, Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update, by the Evidence-based Practice Center Program at the Agency for Healthcare Research and Quality (AHRQ).