Topic Development

This topic was developed in response to a nomination by Dr. William Gunnar, National Director of Surgery (10NC2). Key questions were then developed with input from the topic nominator, the ESP coordinating center, the review team, and the technical expert panel (TEP).

The Key Questions were:

• KQ1: Among adults with CLI, what is the cost-effectiveness of leg bypass compared to endovascular procedures including balloon angioplasty, arterial stents, and atherectomy?
• KQ2: Does the cost-effectiveness of leg bypass compared to endovascular procedures for CLI vary by patient population, setting, or time (short vs long-term)?

The review was registered in PROSPERO: CRD42018106431.

Search Strategy

We conducted searches in PubMed from 1/1/2000-1/16/2019 and Embase from 1/1/2000-1/17/2019. The search used a broad set of terms relating to “limb ischemia” or “endovascular intervention” or “surgical intervention", and utilization measures including “cost-effectiveness". Evidence from studies published prior to the year 2000 were determined to be insufficiently relevant to modern practice. See Appendix A for complete search strategy.

Study Selection

Four team members independently screened the titles of retrieved citations. For titles deemed relevant by at least 1 person, abstracts were then screened independently in duplicate by 4 team members working in pairs. All disagreements were reconciled through group discussion. Full-text review was conducted in duplicate by 2 independent team members, with any disagreements resolved through discussion. Studies were included at either the abstract or the full-text level if they were randomized control trials comparing surgery with endovascular therapy that included and reported separately outcomes for patients with CLI. We also included publications of cost-effectiveness models that compared surgery with endovascular therapy for patients with CLI. Because of the expected paucity of RCTs we also included observational studies. In order to be included, an observational study had to include at least 500 subjects, report comparative data on an effectiveness outcome (such as amputation free survival, mortality, etc) and a cost or utilization outcome (such as cost, length of stay, etc), or be a study of a VA patient population, and be in a context compatible with current US practice. This last category meant that we excluded a few studies that had hospital length of stay data far exceeding current US practice such as Ireland, Finland, and Australia, where both endovascular and surgical groups had hospital length of stay exceeding 30 days.^5–8

Data Abstraction

Data extraction was completed in duplicate. All discrepancies were resolved with full group discussion. We abstracted data on the following: study design, single versus multi-site study, patient characteristics, sample size, comparison, utilization measures, efficacy outcomes, duration of follow-up, and data needed for the Cochrane Risk of Bias tool.

Quality Assessment

Randomized controlled trials were assessed for quality (risk of bias) with the Cochrane Risk of Bias tool.⁹ This tool requires an assessment of whether a study is at high or low (or unknown) risk of bias in 7 domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, and other (See Appendix C). We used the Risk of Bias In Non-randomized Studies – of Interventions (ROBINS-I) for observational studies.¹⁰ This tool requires an assessment of whether a study is at critical, serious, moderate, or low risk of bias (or no information) in 7 domains: confounding, selection bias, bias in measurement classification of interventions, bias due to deviations from intended interventions, bias due to missing data, bias in measurement of outcomes, and bias in selection of the reported result (see Appendix D). Since observational studies are not required to have published an a priori protocol, we operationalized the last domain (bias in selection of the reported result) as requiring that studies report the most common variables.

Data Synthesis and Analysis

Because there was only one randomized control trial, there was no opportunity to conduct meta-analysis of trials. The observational studies were too clinically heterogeneous to support meta-analysis; hence, our synthesis is narrative.

Rating the Body of Evidence

We used the criteria of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group.¹¹ GRADE assessing the certainty of the evidence based of the assessment of the following domains: risk of bias, imprecision, inconsistency, indirectness, and publication bias. This results in categories as follows:

High: We are very confident that the true effect lies close to that of the estimate of the effect.

Moderate: We are moderately confident in the effect estimate. The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different.

Low: Our confidence in the effect estimate is limited. The true effect may be substantially different from the estimate of the effect.

Very low/Insufficient: We have very little confidence in the effect estimate. The true effect is likely to be substantially different from the estimate of effect.

Peer Review

A draft version of the report was reviewed by technical experts and clinical leadership. Reviewer comments and our response are documented in Appendix B.