Topic Refinement and Review Protocol

The topic for this report was nominated in a public process. With input from technical experts, the EPC drafted the initial KQs and, after approval from AHRQ, posted them to a public Web site. The KQs were posted for 4 weeks for public comment. We modified the KQs and the key elements of PICOTS based on these comments and discussion with the TEP.

When the KQs were first written, both the questions and the interventions were stratified by intent of treatment (palliative or curative). However, this stratification seemed clinically inappropriate and potentially confusing because some interventions could be applied to palliate symptoms and to eliminate (i.e., cure) the liver metastases. Thus, the final KQs are distinguished by the population receiving local hepatic therapy. KQs 1 and 2 apply to patients whose CRC is refractory to systemic chemotherapy (i.e., their disease had progressed), and KQs 3 and 4 apply to patients who are receiving local hepatic therapy and systemic chemotherapy. To be consistent with clinical practice, we modified KQs 1 and 2 to include patients with minimal extrahepatic disease. In addition, we categorized the 12 interventions to apply to all KQs, we removed some interventions, and we added SBRT. Finally, we expanded the list of harms to be considered to include elevated alkaline phosphatase, elevated bilirubin, elevated transaminases, liver failure, and rare adverse events that had not been considered originally.

Literature Search Strategy

Inclusion and Exclusion Criteria

Table 3 lists the inclusion/exclusion criteria we selected based on our understanding of the literature, key informant and public comments gathered during the topic refinement phase, input from the TEP, and established principles of systematic review methods.

Table 3

Inclusion and exclusion criteria.

Study Selection

Search results were transferred to EndNote^® and subsequently into DistillerSR (Evidence Partners Inc., Ottawa, Canada) for selection. Using the study selection criteria for screening titles and abstracts, each citation was marked as: (1) eligible for review as full-text articles or (2) ineligible for full-text review. Reasons for article exclusions at this level were not noted. The first-level title-only screening was performed in duplicate. To be excluded, a study needed to be independently excluded by both team members. In cases where there was disagreement, second-level abstract screening was completed by two independent reviewers.

Discrepancies were decided by consensus opinion and a third reviewer was consulted when necessary. All team members were trained using a set of 50 abstracts to ensure uniform application of screening criteria. Full-text review was performed if it was unclear whether the abstract met article-selection criteria.

Full-text articles were reviewed in the same fashion to determine their inclusion in the systematic review. Records of the reason for exclusion for each paper retrieved in full-text, but excluded from the review, were maintained in the DistillerSR database. Although an article may have been excluded for multiple reasons, only the first reason identified was recorded.

Development of Evidence Tables and Data Extraction

Evidence tables were constructed by clinical content experts and staff at the EPC. Tables were designed to provide sufficient information and enable readers to understand the studies and determine their quality. Emphasis was given to data elements essential to our KQs. Evidence table templates were identical for KQ1 and KQ3 and KQ2 and KQ4. The format of our evidence tables was based on examples from prior systematic reviews.

Data extraction was performed directly into tables created in DistillerSR, with elements defined in an accompanying data dictionary. All team members extracted a training set of five articles into evidence tables to ensure uniform extraction procedures and test the utility of the table design. All data extractions were performed in duplicate, with discrepancies resolved by consensus. The full research team met regularly during the period of article extraction to discuss any issues related to the extraction process. Extracted data included patient and treatment characteristics, outcomes related to intervention effectiveness, and information on harms. Harms included specific negative effects, including the narrower definition of adverse effects. Data extraction forms used during this review are presented in Appendix C.

The final evidence tables are presented in their entirety in Appendix D. Studies are presented in the evidence tables by study design, then year of publication alphabetically by the last name of the first author. Abbreviations and acronyms used in the tables are listed as table notes and are presented in Appendix E.

Risk of Bias Assessment of Individual Studies

For the assessment of risk of bias in individual studies, we followed the Methods Guide³⁸ where applicable. Our assessment of risk of bias in the included case-series intervention studies was based on a set of study characteristics proposed by Carey and Boden.⁶⁰ These characteristics include: clearly defined study questions, well-described study population, well-described intervention, use of validated outcome measures, appropriate statistical analyses, well-described results, discussion and conclusion supported by data, and acknowledgement of the funding source. The Carey and Boden assessment tool does not conclude with an overall score of the individual study. We created thresholds for converting the Carey and Boden⁶⁰ risk assessment tool into AHRQ standard quality ratings (good, fair, and poor) to differentiate case-series studies of varied quality. These distinctions are to be used for differentiation within the group of case-series studies, but not for the overall body of evidence described below. The classification into these categories (i.e., good, fair, poor) is distinct for a specific study design. Other study designs are evaluated according to their own strengths and weaknesses.

For a study to be ranked as good quality, each of the Carey and Boden⁶⁰ criteria must have been met. For a fair quality rank, one criterion was not met, and a rank of poor quality was given to studies with more than one criterion not met. These quality ranking forms can be found in Appendix D.

Data Synthesis

Evidence tables were completed for all included studies, and data are presented in summary tables. Evidence is also presented in text organized by outcome and intervention. No direct comparisons are made. We considered whether formal data synthesis (e.g., meta-analysis) would be possible from the set of included studies. Because the literature was so heterogeneous in terms of the populations (e.g., prior treatments, reason for unresectability and number and size of lesions) and interventions (e.g., drugs and dose) studied, we concluded that pooling data would be inappropriate for this review. Thus, all data synthesis is based on qualitative summaries and analyses.

Strength of the Body of Evidence

We graded the strength of the overall body of evidence for overall survival, quality of life, and harms for the four KQs. We used the EPC approach (developed for the EPC program and referenced in the Methods Guide^38,61), which is based on a system developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group.⁶² This system explicitly addresses four required domains: risk of bias, consistency, directness, and precision.

The overall strength of evidence could be graded as “high” (indicating high confidence that the evidence reflects the true effect, and that further research is very unlikely to change our confidence in the estimate of effect); “moderate” (indicating moderate confidence that the evidence reflects the true effect, and that further research may change our confidence in the estimate of effect and may change the estimate); “low” (indicating low confidence that the evidence reflects the true effect, and that further research is likely to change our confidence in the estimate of effect and is likely to change the estimate); or “insufficient” (indicating that evidence is either unavailable or does not permit estimation of an effect).

Two independent reviewers rated all studies on domain scores and resolved disagreements by consensus discussion; the same reviewers also used the domain scores to assign an overall strength of evidence grade. When evidence was available but the effects could not be estimated from the body of evidence, the overall strength of evidence was rated as “insufficient.” If we could estimate comparative effects, we graded the evidence as “low,” indicating our low level of confidence in the estimates. This decision was based in large part on the biases inherent in a literature base comprising case-series studies. In this review, consistency of the body of literature was graded as “not applicable.” The direction of effect cannot be assessed in noncomparative studies; therefore, consistency in the direction of effect across case series cannot be discerned. In the absence of a comparator, we do not know if the observed estimate is better or worse; therefore, we concluded that consistency was not applicable. Directness pertains to the whether the evidence links the interventions directly to a health outcome. Due to the absence of direct comparisons precision will be rated imprecise.