U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Optimal Strategies for the Diagnosis of Acute Pulmonary Embolism: A Health Technology Assessment — Project Protocol [Internet]. Ottawa (ON): Canadian Agency for Drugs and Technologies in Health; 2016 Sep 29. (CADTH Optimal Use Report, No. 6.3a.)

Cover of Optimal Strategies for the Diagnosis of Acute Pulmonary Embolism: A Health Technology Assessment — Project Protocol

Optimal Strategies for the Diagnosis of Acute Pulmonary Embolism: A Health Technology Assessment — Project Protocol [Internet].

Show details

CLINICAL REVIEW

This protocol was written a priori and will be followed throughout the review process. Any deviations from the protocol will be disclosed in the final report and updates will be made to the PROSPERO submission accordingly.

Selection Criteria

The selection criteria for research questions 1 through 3 can be found in Table 1. The clinical care pathway is presented in Appendix 5. For the research question (1) addressed by an overview of SRs, the selection criteria apply to the criteria used by the potentially relevant SRs in identifying primary studies to include. For the research questions (2 and 3) addressed by SRs of primary studies, the selection criteria apply to the population, intervention, comparator, outcomes (PICOs) elements of the individual studies.

Table 1. Selection Criteria for Clinical Research Questions.

Table 1

Selection Criteria for Clinical Research Questions.

Study Design

An informal scoping review of existing HTAs and SRs identified through a MEDLINE and grey literature search was conducted to inform the preparation of this protocol. Due to the breadth of evidence available regarding risk stratification strategies, it was decided that a systematic overview of existing HTAs and SRs would be appropriate to address research question 1. Based on observed heterogeneity in the scope of existing SRs on diagnostic imaging studies and diagnostic algorithms including imaging studies, it was decided that a systematic review of primary studies would be most appropriate to address research questions 2 and 3. To reflect the differences in the chosen methodological approaches, sections of this protocol will be organized by A) risk stratification (research question 1) and B) diagnostic pathways and imaging studies (research questions 2 and 3).

Risk Stratification

For interventions used for risk stratification of patients with suspected PE (research question 1), a systematic overview of SRs available in the literature on the diagnostic test accuracy, comparative clinical utility, and safety of the interventions of interest will be conducted. All SRs identified that meet selection criteria will be included, including SRs with or without a meta-analysis (MA), as well as HTAs that include an SR with or without an MA.

If SRs are available, no primary studies, including those published after the latest search date of the included SRs, will be included. If no published SRs on any given intervention, comparator, or outcome of interest are identified, a SR of primary studies may be conducted. At that time, a primary review protocol will be developed accordingly and become an addendum to this protocol.

Diagnostic Pathways and Diagnostic Imaging Studies

An SR of primary studies on the diagnostic test accuracy, comparative clinical utility, and safety of diagnostic pathways and imaging studies used for PE diagnosis in adult patients being tested for PE will be conducted to address research questions 2 and 3. Randomized controlled and controlled non-randomized clinical studies will be included. Uncontrolled studies will be included only for studies investigating patient harms. A high-level narrative overview of the results of the aforementioned scoping review of SRs will be presented to provide context for the discussion.

Exclusion Criteria

Risk Stratification

To be included for research question 1, SRs must have the term “systematic review” or “meta-analysis” in the title or elsewhere in the text; include a detailed description of comprehensive selection criteria and search methods (i.e., as described in Assessment of Multiple Systematic Reviews [AMSTAR] checklist item 3, search at least two electronic sources, adequately report years searched and databases used, key words and/or MeSH terms, and where feasible, the search strategy provided); assess the quality (or risk of bias) of included studies; and synthesize the findings quantitatively and/or qualitatively. For studies that meet the other criteria but do not perform a quality assessment of the primary included studies, the studies will be included as SRs if they have relevant outcomes or subgroups that are not present in any of the other included SRs, or if they include unique primary studies not reviewed by another SR. In this case, quality assessment of the primary studies will be conducted de novo in duplicate. SRs will be excluded if they do not meet the selection criteria outlined in Table 1, if they are duplicate publications, or if they were published prior to 2006. Multiple publications of the same SR will be excluded unless they provide additional outcomes of interest. Older SRs (based on publication year) identified in the literature search results will be excluded if they are superseded by an updated SR, or if all the included studies in the older SRs are included in newer SRs. However, two SRs that include identical primary studies will be included if they report different outcomes or identical outcomes but present different subgroup analyses. The degree of overlap between SRs with overlapping primary studies will be judged by building a matrix of included studies in the SRs, which will be reported within the results section of a final report. If an SR has unique primary studies, but they are limited to case reports, they will be excluded in the interest of appraising higher-quality evidence and in acknowledgement of design limitations resulting in high risk of bias. Similarly, SRs with unique primary studies that do not meet the inclusion criteria will be excluded if all other studies are included within another SR. A list of excluded studies, with reasons for exclusion after full-text review, will be provided.

Diagnostic Pathways and Diagnostic Imaging Studies

For questions being addressed by a review of primary studies (research questions 2 and 3), studies will be excluded if they do not meet the selection criteria outlined in Table 1, if they are case reports, or if they are duplicate publications. If there are multiple publications of the same study, the less recent will be excluded unless it provides additional information on the outcomes of interest. There is no restriction regarding the duration of time between symptom presentation and assessment, or length of follow-up. Studies will be excluded if they are not published in English or French. Further, conference abstracts, published thesis documents, and evidence that have not been peer-reviewed will not be included.

Screening and Selecting Studies for Inclusion

Two reviewers will independently screen titles and abstracts of all citations retrieved from the literature search, reference lists of identified eligible studies, and any articles identified by content experts. This will be followed by an independent review of the full-text articles selected by the two reviewers, based on the pre-determined selection criteria outlined in Table 1. The two reviewers will compare their selections from the full-text review and resolve disagreements through discussion until consensus is reached, consulting a third reviewer if necessary. A final draft list of included studies will be posted for stakeholder review for 15 business days, and feedback and any additional studies identified for potential inclusion will be reviewed following the above process.

A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart48 will be used to report the study selection process. A full-text screening checklist is reported in Appendix 2.

Data Extraction

Standardized data extraction forms (Appendix 3; Appendix 4) have been designed a priori to document and tabulate all relevant information from included studies. Relevant information includes both descriptive data and results reported in all included studies. Two reviewers will pilot the extraction forms in duplicate among pairs of randomly selected included studies until consistency between reviewers is reached.

Once consistency is reached, data from each included study will be extracted by one reviewer and checked for accuracy by a second reviewer (i.e., the studies will be randomly divided into two independent groups of equal size, one of which will be extracted by one reviewer and the other by another reviewer. Each reviewer will then verify the data extracted from the group of studies they did not extract). Disagreements will be resolved through discussion, involving a third reviewer, if necessary. Data will not be extracted from figures if they do not explicitly provide numerical data. For SRs, primary studies will be consulted for any missing information, to clarify any issues, or to verify extracted data, if necessary. Authors of the studies included in this review will be contacted to provide any missing information or clarify any issues.

Methodological Assessments

Risk Stratification

For question 1, which is being addressed by an overview of SRs, the Risk of Bias in Systematic Reviews (ROBIS) tool,49 designed to assess the risk of bias in SRs of RCTs and non-randomized studies will be used. Although the results of the methodological assessments will not be used to exclude the included SRs, the conclusions and discussion of the final report will focus on the findings of the SRs of higher quality.

Two reviewers will pilot the quality assessment tools on pairs of two randomly chosen studies and, once consistency in assessments is reached, then independently assess the methodological quality of the remaining included studies. Disagreements will be resolved through discussion, involving a third reviewer, if necessary.

Diagnostic Pathways and Diagnostic Imaging Studies

For the questions (research questions 2 and 3) being addressed by an SR of primary studies, clinical RCTs will be assessed using the Cochrane Risk of Bias Tool.50 Clinical non-randomized studies will be assessed using the Risk of Bias Assessment Tool for Nonrandomized Studies (ROBINS-I).51 Studies assessing diagnostic test accuracy will be evaluated used the QUADAS-II instrument.52 The results of the methodological assessments will not be used to exclude primary studies, but the conclusions and discussion of the final report will focus on the findings of the studies of higher quality.

Summary of Evidence

Description of Study Characteristics and Findings

Risk Stratification

A summary of SR characteristics — including the total number of SRs by population, intervention, comparator, outcomes, and study design (PICOS) elements, countries and years of publication, and findings — will be provided. Additionally, information from SRs — including characteristics, numbers, types, and years of publication of primary studies included in the SRs — will be summarized. In the case that more than one SR is included for a given intervention, the comparator(s), and outcome(s) of interest, and any overlap of included studies among SRs will be described and presented (e.g., by preparing a matrix of included studies in the SRs).

Diagnostic Pathways and Diagnostic Imaging Studies

A summary of primary study characteristics — including the total number of studies by PICOS elements, and countries and years of publication — will be provided in the form of tables and a narrative summary.

Description of Methodological Assessments

A narrative summary of the results of methodological assessment for each included study will be provided. Specifically, tables will be developed to present the answers to the questions within the respective assessment tools, with symbol codes to distinguish between classifications. A narrative description of the strengths and limitations of the included studies will be presented within the main text of the report to provide the reader with an overview of the quality of the literature and to highlight any nuances not addressed in the methodological assessment tables.

Narrative Synthesis

Risk Stratification

A narrative synthesis of the results of included SRs will be conducted for research question 1. The findings will be grouped by outcome, with diagnostic test accuracy, clinical utility, and safety outcomes grouped separately. No re-synthesis of the findings from primary studies will be conducted. Results will be represented as reported in SR study reports, including a summary estimate and confidence interval (CI), measure of heterogeneity, and number of studies and participants contributing to each estimate, as available. Tables will be developed to present results by outcome. This is intended to accompany the narrative summary, to ensure consistency of presented information across all included SRs and to facilitate comparisons by the reader. Results will be synthesized by outcome for the overall study population, and also for each subgroup listed in Table 1.

Diagnostic Imaging Studies

In addition to the SR of primary studies described above, a narrative synthesis of the results of the scoping review of SRs on diagnostic imaging studies will be presented to contextualize the findings of the primary study review conducted for questions 2 and 3. The findings will be grouped by outcome, with diagnostic test accuracy, clinical utility, and safety outcomes grouped separately. No re-synthesis of the findings from primary studies will be conducted. Results will be represented as reported in SR study reports, including a summary estimate and CI, measure of heterogeneity, and number of studies and participants contributing to each estimate, as available. Tables will be developed to present results by outcome. This is intended to accompany the narrative summary, to ensure consistency of presented information across all included SRs and to facilitate comparisons by the reader. Results will be narratively synthesized by outcome for the overall study population, and also for each subgroup listed in Table 1.

Statistical Analysis

Data Synthesis Methods

The results of studies included for questions addressed by a SR of primary studies (questions 2 and 3) will be pooled using MA, if appropriate. The decision to pool all studies or subsets of studies will be made after review and exploration of heterogeneity. Clinical and methodological heterogeneity will be assessed in consultation with the clinical experts. This assessment will consider patient and study design factors that might be expected to affect test performance. This will include assessment of heterogeneity of composite reference standards used in the primary studies. If pooling is not appropriate, due to significant clinical heterogeneity, or methodological or statistical heterogeneity that cannot be addressed analytically, the findings will be synthesized narratively.

For each outcome of interest, analysis will be conducted for the overall study population and also for each subgroup listed in Table 1, as the data permit.

Meta-Analysis of Diagnostic Test Accuracy Studies

Based on the scoping review, studies that assessed diagnostic accuracy varied in terms of their applied reference standards, diagnostic pathways, settings, and patient populations. Thus, it is unclear whether pooling will be appropriate. If MA is deemed inappropriate, studies that report on diagnostic accuracy will be reviewed and results will be reported narratively.

Between-study heterogeneity within groups of studies being considered for pooling will be assessed, using graphical presentations including forest plots and plots of sensitivity and specificity in ROC-space, and calculation of between-study variance t2, summary and predictive CIs.53

Reasons for observed heterogeneity will be explored by subgroup or multivariate regression analyses, given the availability of covariate data. Individual comparisons will be summarized separately, and the consistency assessed. Additional sensitivity analyses dealing with study outliers, study size, study quality, study design, and other study- or design-related factors will also be considered to establish the robustness of findings. As some variation in patient population and associated prevalence of PE is anticipated, risk of verification bias as determined during critical appraisal will be assessed in sensitivity analysis. If substantial verification bias is detected, models will be adjusted using the method of de Groot et al.54

There are no established thresholds to determine the appropriateness of pooling of diagnostic testing studies,53 so the findings of the above will be appraised in terms of usefulness in answering our clinical and policy questions. Should we decide that MA is appropriate, we will use the model developed by Rutter and Gatsonis to generate hierarchical summary receiver operating characteristic (HSROC) curves,55 as well as pooled sensitivity, specificity, diagnostic odds ratio (DOR) values, and their 95% CIs. The area under the curve (AUC) will be used as a quantitative measure of the diagnostic accuracy of imaging studies for PE diagnosis, with values closer to 1 indicating better diagnostic performance, and a value closer to 0.5 indicating poor performance.56 Positive and negative likelihood ratios above 10 and below 0.1, respectively, will indicate low misdiagnosis rates.

In the event that we observe substantial variation in reference standards between studies put forth for pooling, particularly involving composites of multiple tests, we will use an extension of the Rutter and Gatsonis model that allows imperfect and composite reference standards.57,58 Unknown parameters will be estimated using a Bayesian approach with non-informative prior distributions, which allow the observed data to dominate the final estimates of sensitivity and specificity. Our models will assume independence of the combined tests acknowledging evidence that suggests the results may be affected if tests show dependency.59,60

Exploration of heterogeneity, plotting, and MA will be conducted using the statistical software R,61 with packages mada62 and HSROC.63

If pooling is not appropriate, a narrative synthesis will include the presentation of findings within summary tables, alongside study and clinical characteristics believed to contribute to heterogeneity, as determined during the exploration of the data. A narrative description will aim to synthesize observed test performance in the absence of an MA.

Meta-Analysis of Primary Clinical Utility and Safety Studies

The clinical utility of risk stratification strategies and imaging studies for PE diagnosis will be based on findings about the benefits (e.g., diagnostic efficiency, influence on choice of treatment and subsequent reduced exposure to imaging harms or harms of unnecessary intervention, and the indirect effect on clinical outcomes), and harms (e.g., failure rate).

Dichotomous outcomes (e.g., mortality) will be summarized, using relative risks and 95% CIs (or odds ratios and 95% CIs, if case-control studies are included). Continuous outcomes will be summarized, using differences in means and 95% CIs, if appropriate. If indicated (e.g., for quality-of-life scales), standard methods for converting between units of measurement will be used, and we will calculate standardized mean differences if possible. For outcomes reported as time-to-event and given available individual patient data in the form of a survival curve or table of events per patients at risk, analyses will be performed, using Kaplan–Meier curves and Cox regression. If studies report adjusted effects measures, the adjusted results in the primary analysis will be used, with the unadjusted result in exploratory analyses presented and comments on any differences between the two. If required measures of variance are not available, variances will be imputed if possible.64 Forest plots will be shown for all individual summary estimates. Findings will be reported as “not statistically significant” if the 95% CI of the overall estimate includes unity for dichotomous data or includes 0 for continuous data.

Between-study heterogeneity within groups of studies being considered for pooling will be assessed, using graphical presentations (including forest plots and plots of outcomes against covariates), and calculations of the I2 and Cochran’s Q test statistic. An I2 ≥ 75% will be interpreted to indicate considerable heterogeneity across studies, as suggested by the Cochrane Handbook for Systematic Reviews of Interventions. Cochran’s Q test statistic — based on chi-squared, where I2 = (Q-degrees of freedom)/Q — will be based on a level of significance of 10%. Clinical and methodological heterogeneity will be assessed in consultation with the clinical experts.

Reasons for observed heterogeneity will be explored by subgroup or multivariate regression analyses, given the availability of the data. Individual contrasts will be summarized separately, and the consistency assessed. Additional sensitivity analyses dealing with study outliers, study size, study quality, study design, and other study- or design-related factors will also be considered to establish the robustness of findings.

If pooling of outcome data is appropriate, summary measures and CIs for the reported outcomes will be reported. Random-effects models will be used. In the event that both randomized and non-randomized studies report on the same outcome, RCTs will be considered separately from non-randomized studies. The influence of study design will be explored in sensitivity analyses. MAs would be carried out using the Cochrane Review Manager software, version 5.3, or using R with package metafor.65

If pooling is not appropriate, a narrative synthesis will include the presentation of findings within summary tables, alongside study and clinical characteristics believed to contribute to heterogeneity, as determined during the exploration of the data. A narrative description will aim to synthesize the direction and size of any observed effects across studies in the absence of a MA and will include an assessment of the likelihood of clinical benefit or harm.

Publication bias will be assessed using visual funnel plots, and tested using Egger’s regression test and Begg’s rank correlation test.66

Network Meta-Analysis

The results of studies assessing the diagnostic accuracy of diagnostic imaging studies for PE diagnosis (questions 2 and 3) will be pooled, using network meta-analysis (NMA), if appropriate. Methods will be further developed in consultation with an expert if it is deemed possible to conduct the NMA. The scope of this analysis is presented in Appendix 6. All possible comparisons between diagnostic imaging studies (interventions) of interest will be evaluated.

Copyright © 2016 CADTH.

The copyright and other intellectual property rights in this document are owned by CADTH and its licensors. These rights are protected by the Canadian Copyright Act and other national and international laws and agreements. Users are permitted to make copies of this document for non-commercial purposes only, provided it is not modified when reproduced and appropriate credit is given to CADTH and its licensors.

Except where otherwise noted, this work is distributed under the terms of a Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International licence (CC BY-NC-ND), a copy of which is available at http://creativecommons.org/licenses/by-nc-nd/4.0/

Bookshelf ID: NBK395894

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (513K)

Other titles in this collection

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...