NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Institute of Medicine (US) Committee on Standards for Systematic Reviews of Comparative Effectiveness Research; Eden J, Levit L, Berg A, et al., editors. Finding What Works in Health Care: Standards for Systematic Reviews. Washington (DC): National Academies Press (US); 2011.
Finding What Works in Health Care: Standards for Systematic Reviews.
Show detailsTABLE F-1Comparison of Chapter 4 Guidance on Conducting Systematic Reviews (SRs) of Comparative Effectiveness Research
| Standards and Elements | Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program | Centre for Reviews and Dissemination (CRD) | The Cochrane Collaboration |
|---|---|---|---|
| 4.1 Use a prespecified method to evaluate the body of evidence | The AHRQ method for evaluating the body of evidence is conceptually similar to the GRADE system (see below). | The planned approach to evaluating the body of evidence should be decided at the outset of the review, depending on the type of question posed and the type of studies that are likely to be available. | Adopts the GRADE system for evaluating the body of evidence. |
4.1.1 For each outcome, systematically assess the following characteristics of the body of evidence:
| Requires the assessment of:
| Quality assessment is likely to consider the following:
| Requires the assessment of:
|
| Reviewers should evaluate the applicability of a body of evidence as part of the assessment of directness. | |||
| Reviewers should evaluate the applicability of a body of evidence separately from directness. | The importance of each of these aspects of quality will depend on the focus and nature of the review. | ||
4.1.2 For bodies of evidence that include observational research, also systematically assess the following characteristics for each outcome:
| The following characteristics should be assessed if they are relevant to a particular SR. They are applied more often to evidence from observational studies than to evidence from randomized controlled trials.
| The quality assessment should be guided by the types of study designs included in the SR. | For bodies of evidence that include observational research, assess the following characteristics for each outcome:
|
| 4.1.3 For each outcome specified in the protocol, use consistent language to characterize the level of confidence in the estimates of the effect of an intervention | The quality of evidence receives a single grade: high, moderate, low, or insufficient. | Not mentioned. | The quality of evidence receives a single grade: high, moderate, low, or very low. |
| 4.2 Conduct a qualitative synthesis | All SRs should include a narrative synthesis. Provides guidance (see below). | All SRs should include a narrative synthesis. Provides guidance (see below). | A narrative synthesis should be used where meta-analysis is not feasible or not sensible. Provides guidance on some elements (see below). |
| 4.2.1 Describe the clinical and methodological characteristics of the included studies, including their size, inclusion or exclusion of important subgroups, timeliness, and other relevant factors | Summarize the available evidence using PICOTS domains in a summary table:
| Provide a clear descriptive summary of the included studies, with details about study type, interventions, number of participants, a summary of participant characteristics, outcomes, and outcome measures. | Review authors should, as a minimum, include the following in the characteristics of included studies table: methods, participants, intervention, and outcomes. Where appropriate, use an extra field to provide information about the funding of each study. |
| 4.2.2 Describe the strengths and limitations of individual studies and patterns across studies | Assess and document decisions on “quality” and applicability of individual studies, including criteria for overall quality assessment. | Recording the strengths and weaknesses of included studies provides an indication of whether the results have been unduly influenced by aspects of study design or conduct. | Whether the synthesis is quantitative or qualitative, methodological limitations are described in detail through presentation of risk of bias tables, through written summaries of risk of bias assessments, and by footnotes in summary of findings tables. |
| 4.2.3 Describe, in plain terms, how flaws in the design or execution of the study (or groups of studies) could bias the results, explaining the reasoning behind these judgments | EPCs describe criteria for assessing risk of bias of individual studies, which, by definition, describes how the study design and execution may bias the results. | Assess the risk of bias in included studies caused by inadequacies in study design, conduct, or analysis that may have led to the treatment effect being over- or underestimated. | Assess risk of bias in all studies in a review irrespective of the anticipated variability in either the results or the validity of the included studies. |
| 4.2.4 Describe the relationships between the characteristics of the individual studies and their reported findings and patterns across studies | EPCs should explore heterogeneity of findings. They should prespecify subanalyses or characteristics by which they analyze heterogeneity, whether for methodologic heterogeneity or clinical heterogeneity. | Provide an analysis of the relationships within and between studies. | Organizing the studies into groupings or clusters is encouraged (e.g., by intervention type, population groups, setting, etc.). |
| 4.2.5 Discuss the relevance of individual studies to the populations, comparisons, cointerventions, settings, and outcomes or measures of interest | EPCs should describe the limitations of applicability of a body of evidence within the PICOS structure. | Not mentioned. | Not mentioned. |
| 4.3 Decide if, in addition to a qualitative analysis, the systemic review will include a quantitative analysis (meta-analysis) | Meta-analysis is appropriate if combining studies will give a meaningful answer to a well-formulated research question. | The approach to quantitative synthesis should be decided at the outset of the review. | Describe why a meta-analysis is appropriate. The choice of meta-analysis method should be stated, including whether a fixed-effect or a random-effects model is used. |
| Meta-analysis is not always possible or sensible. The type of synthesis depends on the type of question posed and the type of studies that are available. Initial descriptive phase of synthesis will be helpful in confirming that studies are similar and reliable enough to synthesize and that it is appropriate to pool results. | |||
| 4.3.1 Explain why a pooled estimate might be useful to decision makers | Authors should explain the reason a combined estimate might be useful to decision makers. | Not mentioned. | Not mentioned. |
| 4.4 If conducting a meta-analysis, then do the following: | Provides guidance on conducting a meta-analysis (see below). | Provides guidance on conducting a meta-analysis (see below). | Provides guidance on conducting a meta-analysis (see below). |
| 4.4.1 Use expert methodologists to develop, execute, and peer review the meta-analyses | Review team must include an individual with statistical expertise. A peer reviewer with statistical expertise should be invited as appropriate. | The review team should ideally include expertise in statistics. The team may wish to seek advice from methodological experts formally through an advisory group, or informally. | Review teams must include, or have access to, expertise in systematic review methodology (including statistical expertise). |
| 4.4.2 Address the heterogeneity among study effects | Evaluate the amount of heterogeneity for each meta-analysis. Explore statistical heterogeneity using subgroup analysis or meta-regression or sensitivity analyses. | Variation in results across studies should be investigated informally by visual examination of the forest plot, tested using chi square test or Q statistic, quantified using the I squared statistic. If statistical heterogeneity is observed, then the possible reasons for differences should be explored. The influence of patient-level characteristics or issues related to equity can also be explored through subgroup analyses, meta-regression, or other modeling approaches. | It is important to consider to what extent the results of studies are consistent. A statistical test for heterogeneity is available, but a useful statistic for quantifying inconsistency is I2. It is clearly of interest to determine the causes of heterogeneity among results of studies. However, most Cochrane reviews do not have enough studies to allow the reliable investigation of the reasons for heterogeneity. |
| 4.4.3 Accompany all estimates with measures of statistical uncertainty | Appropriate measures of variance should be included with point estimates from meta-analyses. | Results should be expressed as point estimates together with associated confidence intervals and exact p-values. | Results should always be accompanied by a measure of uncertainty, such as a 95% confidence interval. |
| 4.4.4 Assess the sensitivity of conclusions to changes in the protocol, assumptions, and study selection (sensitivity analysis) | Sensitivity analysis should be conducted to investigate the robustness of the results. | Sensitivity analyses should be used to explore the robustness of the main meta-analysis by repeating the analyses after having made some changes to the data or methods. | Sensitivity analyses should be used to examine whether overall findings are robust to potentially influential decisions. |
NOTES: Some information on AHRQ-, CRD-, and Cochrane-recommended methods was provided via personal communication with Stephanie Chang, EPC Program Task Order Officer, AHRQ (October 5, 2010); Lesley Stewart, Director, CRD (October 14, 2010); and Julian Higgins, Senior Statistician, MRC Biostatistics Unit, Institute of Public Health, University of Cambridge (October 4, 2010). The order of the standards does not indicate the sequence in which they are carried out.
REFERENCES
- Atkins, D., S. Chang, G. Gartlehner, D. I. Buckley, E. P. Whitlock, E. Berliner, and D. Matchar. 2010. Assessing the applicability of studies when comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www
.effectivehealthcare .ahrq.gov/index .cfm/search-for-guides-reviews-and-reports /?productid=603&pageaction=displayproduct (accessed January 19, 2011). - Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):502–512. [PubMed: 18823754]
- CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care . York, UK: York Publishing Services, Ltd.
- Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L. Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A. Trikalinos. 2010. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www
.effectivehealthcare .ahrq.gov/index .cfm/search-for-guides-reviews-and-reports /?pageaction=displayProduct&productID=554 (accessed January 19, 2011). - Helfand, M., and H. Balshem. 2010. AHRQ series paper 2: Principles for developing guidance: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):484–490. [PubMed: 19716268]
- Higgins, J. P. T., editor; , and S. Green, editor. , eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons.
- Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www
.effectivehealthcare .ahrq.gov/index .cfm/search-for-guides-reviews-and-reports /?pageaction=displayProduct&productID=454 (accessed January 19, 2011). - Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ series paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):513–523. [PubMed: 19595577]
- Relevo, R., and H. Balshem. 2011. Finding evidence for comparing medical interventions. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www
.effectivehealthcare .ahrq.gov/index .cfm/search-for-guides-reviews-and-reports /?pageaction=displayProduct&productID=605 (accessed January 19, 2011). - Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ series paper 1: Comparing medical interventions: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):481–483. [PubMed: 18834715]
- White, C. M., S. Ip, M. McPheeters, T. S. Carey, R. Chou, K. N. Lohr, K. Robinson, K. McDonald, and E. Whitlock. 2009. Using existing systematic reviews to replace de novo processes in CERs. In Methods guide for comparative effectiveness reviews, edited by Agency for Healthcare Research and Quality. http://www
.effective-healthcare .ahrq.gov/index .cfm/search-for-guides-reviews-and-reports /?pageaction =displayProduct&productID=329 (accessed January 19, 2011). - Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ series paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health Care Program. Journal of Clinical Epidemiology 63(5):491–501. [PubMed: 19540721]
- PubMedLinks to PubMed
- Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence -...Expert Guidance for Chapter 4: Standards for Synthesizing the Body of Evidence - Finding What Works in Health Care
Your browsing activity is empty.
Activity recording is turned off.
See more...