Structured Abstract

Contents

• Preface
• Acknowledgments
• Steering Committee
• Peer Reviewers
• Executive Summary
• Introduction
  • Quality and Risk of Bias Assessment in Systematic Reviews
  • Goal and Objective
• Methods
  • Steering Committee
  • General Approach
  • Risk of Bias and Randomized Controlled Trials
  • Newcastle-Ottawa Scale and Cohort Studies
• Results
  • Overview
  • Risk of Bias and Randomized Controlled Trials
  • Newcastle-Ottawa Scale and Cohort Studies
• Summary and Discussion
  • Key Points
  • Discussion
  • Implications for Practice
  • Future Directions
  • Strengths and Limitations
  • Conclusions
• References
• Abbreviations
• Appendixes
  • Appendix A Sample of Randomized Controlled Trials
  • Appendix B Guidelines for Risk of Bias Assessments
  • Appendix C Variables for Data Extraction from Randomized Controlled Trials
  • Appendix D Meta-Analyses and Cohort Studies Used for NOS Assessments
  • Appendix E Decision Rules for Application of the Newcastle-Ottawa Scale
  • Appendix F Supplementary Information for NOS Assessments
  • Appendix G Description of Randomized Controlled Trials
  • Appendix H Data Used for Kappa Calculations
  • Appendix I Inter-rater Reliability on Risk of Bias Assessments, by Domain and Study-level Variable With Confidence Intervals
  • Appendix J Sources of Disagreement Across Consensus Assessments Using the Risk of Bias Tool

Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services¹, . Contract No. 290-2007-10021-I. Prepared by: University of Alberta Evidence-based Practice Center, Edmonton, Alberta, Canada

Suggested citation:

Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM. Validity and inter-rater reliability testing of quality assessment instruments. (Prepared by the University of Alberta Evidence-based Practice Center under Contract No. 290-2007-10021-I.) AHRQ Publication No. 12-EHC039-EF. Rockville, MD: Agency for Healthcare Research and Quality. March 2012. www.effectivehealthcare.ahrq.gov/reports/final.cfm.

This report is based on research conducted by the University of Alberta Evidence-based Practice Center under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10021-1). The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

The information in this report is intended to help clinicians, employers, policymakers, and others make informed decisions about the provision of health care services. This report is intended as a reference and not as a substitute for clinical judgment.

This report may be used, in whole or in part, as the basis for the development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products or actions may not be stated or implied.

None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.

1

540 Gaither Road, Rockville, MD 20850; www​.ahrq.gov