Background

The accuracy of non-invasive methods for measuring carotid artery stenosis has received a great deal of attention. Because of the small but persistent complication rate associated with carotid angiography, investigators have worked to establish non-invasive techniques and protocols that can accurately predict high-grade carotid stenosis and thereby reduce or eliminate the need for invasive testing in the selection of candidates for carotid endarterectomy. The non-invasive methods that are most commonly used and that consistently have had the highest accuracy are carotid ultrasonography using Doppler or duplex methods and magnetic resonance angiography. ²³¹

Findings

We identified one good-quality systematic review of studies assessing the accuracy of non-invasive measurement of carotid artery stenosis. ²³¹ From 568 articles retrieved from electronic and manual searches, the authors reviewed 70 studies, published between 1977 and 1993, that allowed quantification of the operating characteristics of B-mode ultrasonography, carotid Doppler, carotid duplex, magnetic resonance angiography, supraorbital Doppler, and oculoplethysmography, as compared to a reference standard of conventional or digital-subtraction carotid angiography. Pooled results demonstrated the superiority of carotid Doppler, carotid duplex, and MRA over the other three non-invasive methods. The sensitivities of the three superior tests ranged from 0.83 to 0.86, and the specificities from 0.89 to 0.94, with narrow 95 percent confidence intervals. Receiver operating characteristic (ROC) curves demonstrated the equivalence of the three methods.

Although the methodological quality of this meta-analysis was high, we had two major concerns about the applicability of its findings. First, technical advances, particularly for the relatively new technology of MRA, may have made the findings of earlier studies obsolete. Second, most of the studies included in the meta-analysis were subject to verification bias. This bias occurs in studies of diagnostic tests when the results of the test under study are used in determining which patients are submitted for testing with the reference standard. An example is a study examining a retrospective sample of all patients who underwent both carotid duplex examination and carotid angiography during a specified time period, in a setting where carotid duplex examination is routinely used to select patients for angiography. Although all patients in the sample underwent both the test under study and the reference standard, most patients with a negative duplex examination have already been excluded because they were not referred for angiography. Excluding negative duplex examinations from the sample artificially reduces the number of both true and false negative tests, thereby inflating sensitivity and reducing specificity. The impact of this bias on estimates of test accuracy can be significant.^232-234 In one study, correction for verification bias diminished sensitivity from 76 to 58 percent and increased specificity from 85 to 92 percent. ²³³

We sought to determine whether the results of more recent studies and studies without verification bias differed from the studies included in the meta-analysis by Blakeley et al. ²³¹ We therefore retrieved studies of the accuracy of non-invasive measurement of carotid artery stenosis published after 1993 and included only studies in which verification bias did not appear to be present. We also used the exclusion criteria employed by Blakeley et al. in their meta-analysis, which we felt represented a minimum standard for studies of non-invasive carotid testing. Studies were excluded "if 1) results from the test used were not compared with the results of conventional carotid angiography or intra-arterial digital subtraction carotid angiography; 2) the angiographic results were not separated to allow for specific identification of occluded arteries; or 3) the reference standard test results could not be classified into a contingency table according to degree of stenosis." ²³¹

Carotid Ultrasound

We identified one good- and four fair-quality studies of carotid Doppler or duplex^{21, 232, 233, 235, 236} and five studies of MRA^{235, 237-240} -- one good, three fair, and one poor -- that met criteria for inclusion. We found that the operating characteristics of these studies were substantially lower than the pooled results in the previous meta-analysis. These studies were reviewed along with the three studies of CUS^241-243 -- one of poor and two of fair quality -- and one poor-quality study of MRA ²⁴⁴ included in the meta-analysis by Blakeley et al. that were not affected by verification bias (Evidence Table 8).

Some studies of the accuracy of CUS used several different criteria, typically based on flow velocity, to diagnose and grade carotid stenosis. When multiple criteria were used, we report the criterion that provided the greatest overall accuracy.

In the largest and only good-quality study of CUS, 1,011 patients enrolled in the NASCET were studied with carotid ultrasound and selective carotid angiography. ²³² Patients were referred for the study from 50 academic medical centers throughout North America. Patients were eligible for the study if they had at least 30 percent ipsilateral stenosis by angiography. Carotid ultrasound was performed at the time of angiography but was not used for the purpose of selecting patients for entry into the study. The authors reported on several different Doppler criteria used to diagnose stenosis, all of which aimed to define stenosis of 70 percent or greater, and all of which gave similar results. The overall sensitivity and specificity of CUS were 69 and 68 percent, respectively.

At least four factors may explain the lower accuracy in this study as compared to others. First, because patients were selected for study on the basis of angiographic rather than CUS results, there was little potential for verification bias, which may explain the lower sensitivity in this study as compared with others. Second, patients with less than 30 percent stenosis were not examined. This may have diminished the number of true negative studies and thereby reduced specificity. Third, this study was conducted across multiple centers. Favorable estimates of test accuracy in the literature may result from a tendency for only centers demonstrating high accuracy to publish their results. This hypothesis was validated by investigators in the Asymptomatic Carotid Atherosclerosis Study, who found that among 37 participating centers asked to submit data on the accuracy of CUS for quality assurance purposes, the sensitivity of CUS required to achieve a positive predictive value of 90 percent ranged from 4 percent to 97 percent. ³¹ Finally, carotid duplex as performed in the NASCET study involved conventional rather than color-flow imaging. It is possible that the newer method of color-flow duplex is more accurate than the older method used in NASCET. One study comparing the two methods found a trend toward greater accuracy with color-flow imaging, ²⁴⁵ while another study found the two methods to be equivalent. ²⁴⁶ In the two included studies reporting the accuracy of color-flow duplex in detecting 70 percent or greater stenosis, sensitivities were 88 and 93 percent, and specificities were 84 and 93 percent.^{21, 235}

The NASCET was the only study to report the interobserver reliability of CUS. One neurosonographer reviewed a subset of 102 videotaped CUS studies from multiple centers, using different techniques for grading stenosis. Agreement between the study sonographer and the readings reported from the local centers was generally very high (kappa 0.79 to 0.98), particularly for the most accurate method of grading stenosis (IC PSV/CC PSV ratio, kappa 0.92 to 0.97). ²³² It should be noted, however, that the reliability of CUS across different operators was not assessed. Differences in skill and technique across operators are likely to be the greatest source of variation in the use of CUS.

Table 9 compares the pooled sensitivity and specificity of CUS in our meta-analysis with results from the meta-analysis by Blakeley et al., which included studies affected by verification bias. We defined positive tests in two ways: angiographic stenoses of 50 percent or more and 70 percent or more as defined by the NASCET method. These represent the thresholds for moderate and severe carotid stenosis, conditions for which the benefits of endarterectomy are quantifiable. We combined studies that reported using a 75 percent cutoff for defining a positive stenosis with those that used a 70 percent cutoff. When studies used the ECST or CC rather than NASCET method for grading angiographic stenosis, we converted the degree of stenosis to its NASCET equivalent, according to a published conversion formula (e.g., 70 percent stenosis by ECST or CC method = 50 percent stenosis by NASCET method). ⁴⁴

Table

Table 9. Pooled accuracy of carotid ultrasound.

It should be noted that the sensitivities and specificities across the eight included studies of CUS were significantly heterogeneous. Pooled results may therefore be of questionable value. We have included them in the table for the purpose of comparison with the results of Blakeley et al. ²³¹ In their meta-analysis, Blakeley et al. also pooled results of studies for which statistical testing indicated significant heterogeneity. We therefore advise caution in interpreting these estimates, which we use only for the sake of comparison.

Pooled estimates of sensitivity and specificity were lower than those reported by Blakeley et al., particularly for the 70 percent stenosis cutoff. When the NASCET study was excluded, accuracy increased. We had expected that the accuracy of CUS in the studies we reviewed might be higher than in those reviewed by Blakeley et al., as a result of technical advances in ultrasound and increasing operator skill as the technology becomes more mainstream over time. The fact that we did not observe higher accuracy is most likely a reflection of our exclusion of studies affected by verification bias. We also expected that excluding studies with verification bias would decrease overall sensitivity but increase specificity. The fact that both sensitivity and specificity in our analysis were lower than those found by Blakeley may indicate that studies with verification bias also suffer from other biases that inflate accuracy.

Magnetic Resonance Angiography

We reviewed six studies of the accuracy of MRA that appeared not to be affected by verification bias (Evidence Table 9).^{235, 237-240, 244} In four of these studies, CUS was explicitly used to select patients for study.^{237-239, 244} While this does not introduce verification bias per se, because the test under study -- MRA -- was not used to determine whether or not a patient underwent the reference standard test, more patients with high-grade stenosis and fewer patients with minimal or no stenosis may have been selected as a result of pre-screening with CUS. This would increase the proportion of true and false positives and decrease the proportion of true and false negatives, inflating sensitivity and decreasing specificity.

Three of the studies excluded patients with indeterminate results,^237-239 and in one study, several included cases were inexplicably missing from the analysis. ²³⁹ We contacted the authors of this study but did not receive a clarifying response. Missing or excluded results may have substantively affected estimates of accuracy in two studies.^{238, 239}

Pooled estimates of accuracy in the six reviewed studies are reported in Table 10, along with estimates from Blakeley et al. In this analysis we combined 50 and 60 percent stenosis cutoffs and included one cutoff of 80 percent in the group reporting accuracy at 70 percent stenosis.

Table

Table 10. Pooled accuracy of magnetic resonance angiography.

Three studies assessed accuracy at detecting moderate stenosis (50 or 60 percent or more by NASCET criteria).^{237, 239, 244} Two of the studies were of poor quality and used only two-dimensional time-of-flight (2D TOF) MRA. Most studies suggest that 2D TOF is less accurate than the more frequently studied combination of 2D and 3D TOF, or 3D TOF used alone.

Studies assessing MRA accuracy for detecting severe stenosis (70 or 80 percent or more) demonstrated relatively high sensitivity and specificity using 3D TOF imaging, either alone or in combination with 2D TOF.^{235, 238, 240} Specificity in the single study using 2D TOF alone was substantially lower. ²⁴⁴ Interobserver reliability of MRA was reported in three studies and was reasonably high (kappa 0.69 to 0.93).^{235, 237, 238}

It should be noted that although we aimed to reduce the influence of verification bias in our review, our results may reflect optimistic estimates of accuracy for MRA. Because MRA is a developing technology, it is likely that investigators reporting on its operating characteristics are those that have the greatest experience and skill in using this technology. Thus, even if unbiased, these estimates of accuracy may not reflect the overall accuracy of MRA in clinical practice.

Supplemental Analyses

Summary Receiver Operating Characteristic Curves

As discussed earlier in Echocardiography Results, variation in sensitivity and specificity of the same diagnostic test may result from the choice of different diagnostic thresholds. This is particularly relevant for CUS. The diagnosis of carotid stenosis using CUS is typically based on the velocity of blood flow at different sites within the carotid artery and during different phases of the cardiac cycle. Many different velocity criteria are used, ²³ and it appears that optimal criteria may vary by equipment and clinical site.^27-30 Because variation in the diagnostic accuracy of CUS often reflects differences in diagnostic thresholds, Summary Receiver Operating Characteristic (SROC) curves provide more appropriate summaries of CUS accuracy than do pooled estimates of average sensitivity and specificity.

The SROC curve for CUS, constructed from the results of studies tabulated in Evidence Table 8, is shown in Figure 7, part A. Most of the studies cluster around the ROC curve. The NASCET study, ²³² however, is a prominent outlier, mainly because of lower specificity than that observed in other studies with similar sensitivity. This may have resulted from the exclusion in NASCET of patients with carotid stenosis of 30 percent or less. Exclusion of patients with absent or minimal stenosis likely decreased the number of true negative CUS tests and thereby diminished specificity. Table 11 shows the sensitivity and specificity of CUS for detecting 50 percent and 70 percent stenosis, both at the point of maximal accuracy on the SROC curves for those stenosis cutoffs (see Appendix F), and at varying diagnostic thresholds. Maximal accuracy as derived from the SROC curves is generally higher than the pooled averages of sensitivity and specificity (Table 9). The estimates of test accuracy across varying thresholds quantitatively demonstrate the tradeoff between sensitivity and specificity depicted in the ROC curves. To achieve a sensitivity of 95 percent or higher, one must in most cases accept a substantial fall in specificity, and vice versa.

Figure

Figure 7. Summary receiver operating characteristic (ROC) curves for the diagnosis of carotid artery stenosis using carotid ultrasound (CUS), magnetic resonance angiography (MRA), or both. Summary ROC curves for (more...)

Table

Table 11. Aggregate operating characteristics of carotid ultrasound at maximal accuracy and varying diagnostic thresholds.

The SROC curve for MRA (Figure 7, part B) shows a relatively wide range of specificity. This may have occurred because of the use of CUS in some studies to select patients for MRA. If only patients with a positive CUS were included, there would likely be fewer true negative MRA studies, which would limit specificity. Nevertheless, the SROC curve was flat over most of the range of specificities, indicating that studies with low specificity did not have a large impact on the shape of the SROC curve. The single outlier study with the lowest sensitivity was a poor-quality study that included only patients with positive CUS and used only 2D-TOF MRA imaging. Table 12 shows the sensitivity and specificity of MRA for detecting 70 percent stenosis, both at the point of maximal accuracy on the SROC curve for this stenosis cutoff (see Appendix F), and at varying diagnostic thresholds. We did not calculate these estimates for 50 percent stenosis because only two studies examined this cutoff, and one of them was of poor quality and used only 2D-TOF imaging.

Table

Table 12. Aggregate operating characteristics of magnetic resonance angiography at maximal accuracy and varying diagnostic thresholds.

The SROC curve for combined CUS and MRA (Figure 7, part C) shows a narrow range of high sensitivity, with a wider range of specificity. As with analysis of MRA alone, the wider array of specificity may have occurred because of the inclusion criteria of a positive CUS in some studies. Similar to the curve for MRA alone, the SROC curve for CUS plus MRA was flat over most of the range of specificities, indicating that studies with low specificity did not have a large impact on the shape of the SROC curve. All of the studies fell on or very near the curve; there were no outliers. Table 13 shows the sensitivity and specificity of combined CUS and MRA for detecting 70 percent stenosis, both at the point of maximal accuracy on the SROC curve for this stenosis cutoff (see Appendix F), and at varying diagnostic thresholds. We did not calculate these estimates for 50 percent stenosis because only three studies examined this cutoff, and two of them were of poor quality.

Table

Table 13. Aggregate operating characteristics of combined carotid ultrasound and magnetic resonance angiography at maximal accuracy and varying diagnostic thresholds.

ROC curves can be used to statistically compare the accuracy of different tests. However, when the tests being compared are applied to different patient populations in different settings, one must be cautious in drawing conclusions from statistical comparisons. ¹⁷⁰ In addition, studies of MRA were generally of poorer quality than those of CUS, and studies of combined CUS and MRA were generally of poor quality and hampered by potential biases. We therefore conclude that the evidence regarding the accuracy of the noninvasive carotid imaging strategies discussed is insufficient to allow comparisons across tests.

Outcomes of Diagnostic Testing

In order to assess the potential outcomes of different testing strategies, we calculated the number of patients in a hypothetical cohort of 1,000 patients presenting with stroke or TIA who would undergo carotid angiography and CEA under several testing strategies (Table 14). In this analysis we varied the prevalence of carotid stenosis, defined as 70 to 99 percent narrowing by NASCET criteria, from 10 to 50 percent and assumed that patients with 70 to 99 percent stenosis of the ipsilateral carotid artery by angiography would be appropriate for CEA. We assumed that CEA would be inappropriate for patients with less than 70 percent stenosis, although we acknowledge that many patients with "false positive" tests would have moderate degrees of carotid stenosis (50 to 69 percent) and might receive some benefit from surgery. ²²⁹

Table

Table 14. Outcomes of diagnostic testing for carotid stenosis in 1000 hypothetical patients with stroke: effect of varying prevalence.

We assumed a sensitivity and specificity of 94 and 84 percent, respectively, for CUS, which represent the maximal accuracy from the SROC curve excluding the NASCET study (Table 11). We used this favorable estimate of accuracy for CUS to allow a more fair comparison with MRA, for which the generally poorer quality of studies and the lack of large, multicenter studies like the NASCET likely provides a higher estimate of accuracy than would be observed in the community. For MRA, the sensitivity and specificity were 92 and 97 percent, as derived from the SROC curve for MRA (Table 12). Finally, for combined CUS and MRA, sensitivity was assumed to be 95 percent and specificity 98 percent (Table 13). The rate of nonconcordant CUS and MRA was assumed to be 18.2 percent, the average rate across the 11 studies of this diagnostic strategy. Although nonconcordance between CUS and MRA may vary with prevalence of stenosis, there was no clear pattern of correlation between prevalence of stenosis and nonconcordance across the studies we reviewed. We therefore assumed the same rate of test nonconcordance across varying degrees of stenosis prevalence.

The analysis demonstrates that at a 10 percent prevalence of stenosis, when CUS is used alone, the number of patients who would undergo CEA for less than 70 percent stenosis would approach 15 percent of the total cohort. If patients with positive CUS proceeded to cerebral angiography, nearly a quarter of patients would undergo angiography, but because angiography is considered the gold standard for selecting patients for surgery, no patients would inappropriately undergo CEA. The risk of harm with CEA is substantially greater than that of angiography, making angiographic confirmation of CUS the safer strategy at low prevalence. As prevalence rises, more patients would undergo angiography, and fewer would undergo CEA inappropriately. The same pattern is observed for MRA, though because of the assumed superior specificity, fewer patients at each level of prevalence would undergo inappropriate CEA.

Assuming a rate of nonconcordance between CUS and MRA of 18 percent, CUS plus MRA is inferior to MRA alone at low prevalence of stenosis. When patients with positive MRA are referred for angiography, approximately 12 percent undergo angiography, and none undergo CEA inappropriately. With CUS and MRA combined, 18 percent (all with nonconcordant tests) undergo angiography, and 1 percent (those for whom both tests are falsely positive) undergo CEA inappropriately. As prevalence rises, combined CUS and MRA becomes a more favorable strategy in comparison to the others.

Summary

Despite numerous studies of the accuracy of noninvasive carotid imaging, relatively few have been conducted in which all patients undergoing noninvasive tests also undergo diagnostic confirmation with cerebral angiography. The lack of diagnostic verification in these studies creates biased estimates of sensitivity and specificity. Studies can adjust for this bias by angiographically studying a random sample of subjects with negative noninvasive tests. We reviewed studies of CUS and MRA accuracy that either had no obvious or likely verification bias or that adjusted for this bias.

It is clear from the literature that the accuracy of CUS in diagnosing carotid stenosis varies substantially across centers. It is likely that published reports of the accuracy of CUS from single centers overestimate the accuracy in most settings. This has two important implications. First, it may be inappropriate for individual practitioners or medical centers to assume that the accuracy of CUS in their practices is equivalent to published figures. Second, it is clear that there is potential for CUS to be highly accurate. The sensitivity and specificity of CUS estimated from SROC curves constructed from the results of eight predominantly fair-quality studies were 80 and 91 percent, respectively, for moderate or greater (> 50 percent) stenosis, and 75 and 87 percent for severe (> 70 percent) stenosis. When the largest and only good-quality study was excluded, sensitivity and specificity for severe stenosis rose to 94 and 84 percent. The lower accuracy in the largest study than in other studies may have been due to the use of conventional rather than color-flow duplex imaging, but may also have been due to the representation of multiple centers. Reports from single centers may provide biased estimates of accuracy, as those centers finding low accuracy may choose not to submit their results for publication.

Whether the accuracy of MRA varies by center is not clear. There have not been multicenter studies of MRA. Published data, excluding studies with obvious or likely verification bias, suggest a sensitivity and specificity of 92 and 97 percent for detecting severe stenosis. However, studies of MRA were generally of fair to poor quality. As with CUS, it is possible that centers publishing their accuracy data are not representative of all users of MRA. Until there are more high-quality data on the accuracy of MRA, current estimates of MRA accuracy in measuring carotid stenosis must be interpreted cautiously.

All studies of the accuracy of CUS and MRA in combination that we identified were biased by incomplete verification. In the majority of these studies, sensitivity was 100 percent. However, these studies were generally of poor quality. The specificity of combined CUS and MRA was variable, ranging from 69 to 100 percent. The estimated sensitivity and specificity of combined CUS and MRA for detecting severe stenosis were 95 and 98 percent, respectively. In approximately 18 percent of patients, the results of CUS and MRA in detecting severe stenosis were discordant.

Findings of Individual Studies

Because the review ²⁴⁸ received a poor quality rating, we carried out a systematic review of prospective studies reporting complication rates of cerebral angiography in patients with symptomatic cerebrovascular disease. We limited our search to prospective studies because of the lower complication rates in the retrospective studies reviewed by Hankey. ²⁴⁸

We identified a total of eleven studies with original data that met our inclusion criteria (see Evidence Table 11), including all eight prospective studies reviewed by Hankey.^249-256 For one study, we substituted data from an earlier report ²⁵⁷ published by the same author ²⁵⁰ on the same population, because the complication rate in symptomatic patients could be calculated only in the earlier study. We identified an additional three prospective studies on conventional angiography or intra-arterial digital subtraction angiography^258-260 published since 1990. We identified no other studies published prior to 1990 in which the rate of complications in symptomatic patients could be calculated. Because differences in major complication rates between different angiographic contrast agents have not been demonstrated, we did not separate the results of studies using different agents. ²⁶¹ The outcomes consistently reported across studies were stroke (typically not classified by severity) and death.

We excluded studies on IV-DSA ²⁶² or other outdated methods such as direct carotid or brachial puncture.^{263, 264} Two studies that examined the same populations as other included studies^{265, 266} were excluded, as were two studies in which the complication rates for different arteriographic procedures (e.g., cerebral, vertebral, or aortic angiography) were not reported.^{267, 268} In three prospective studies reporting complication rates of cerebral angiography, the rates in symptomatic patients could not be calculated separately from the rates in asymptomatic patients, and these studies were excluded.^269-271

All of the studies included in our review were observational, except for one by Skalpe, ²⁵⁷ which was a small randomized trial of different types of contrast used during cerebral angiography (no significant difference in complications was found). We abstracted the year of publication, setting, author departmental affiliations (only radiology or other), rates of death, and rates of combined death and stroke (Evidence Table 11). We assigned an overall quality rating to each study using eight ratings criteria (each individually abstracted) derived from USPSTF quality ratings criteria for cohort studies, adapted for studies looking at complication rates (see Methods). By convention, "adequate" post-angiography duration of followup was defined as 72 hours. ²⁴⁸ Non-independent ascertainment was defined as ascertainment of complications by a non-radiologist.

Four studies were rated as having poor quality,^{249, 252, 254, 257} four fair quality,^{253, 255, 260} and three good quality.^{251, 258, 259} The three studies rated as having good quality included a total of 1,219 symptomatic patients who underwent cerebral angiography. The only methodological concerns identified in these studies were unclear independent ascertainment ²⁵⁸ and unclear duration of followup. ²⁵¹ The study by Hankey ²⁵⁹ appeared to be of overall highest quality, and no major methodological limitations were identified. In this study, the rate of cerebrovascular accident or death was 1.3 percent (5/382). No deaths were reported in any of these studies. The rate of stroke or death ranged from 0 percent (0/637) ²⁵¹ to 4 percent (8/200) ²⁵⁸ in the other studies rated as having good quality.

In studies rated poor, major areas of concern included inadequate characterization of the study population,^{249, 252} unclear selection methods,^{252, 254} unclear techniques to ascertain complication rates,^{252, 254} unclear length of followup,^{252, 254, 257} biased allocation of patients to more experienced angiographer, ²⁵⁶ inadequate description of excluded patients, ²⁵⁷ exclusion of patients undergoing angiography outside "regular working hours," ²⁵⁷ and lack of statistical analysis of potential confounders or risk factors.^{249, 252, 254, 257} The major methodological areas of concern in studies rated fair quality were inadequate analysis of potential confounders,^{253, 260} unclear ascertainment methods,^{253, 255, 260} and unclear duration of followup.^{253, 260} Rates of stroke or death in these studies ranged from 0 percent^{252, 253} to 5.7 percent (13/230). ²⁵⁶ Only two deaths were reported.

Major complications other than stroke and death (e.g., non-fatal cardiovascular and pulmonary events) were infrequently and inconsistently reported. We did not review local complications that typically do not result in long-term sequelae (e.g., hematoma, dysphagia).

Pooled Findings

We did not find significant heterogeneity (chi-square=6.65, df 9, p=0.67) in the rates of death reported from all 10 studies that reported mortality separately. The pooled rate of death was 0.02 percent (2/3074; 95 percent CI, 0 to 0.1 percent).

There was significant heterogeneity between rates of combined stroke or death from all studies (chi-square=24.8, df 8, p=0.002). In order to evaluate potential explanatory factors for the observed heterogeneity, we stratified results by author departmental affiliation (radiology versus non-radiology), study quality (good quality versus fair or poor quality), and year of publication (prior to 1990 versus 1990 or later). Significant heterogeneity remained after stratifying the results by each of these characteristics. Because one study ²⁵⁶ reported an unusually high rate of major periprocedural complications (13/230, 5.7 percent), we performed a test of heterogeneity after excluding this study, and still found significant heterogeneity (chi-square=24.8, df 8, p=0.002). Because of these findings, we did not pool the results of the studies for rates of combined stroke or death.

Risk Factors for Complications

There are few data to assess clinical, demographic, or other factors associated with higher complication rates. Three included studies^{248, 251, 258} performed detailed statistical analysis of potential confounders and risk factors. In a univariate analysis, ²⁵¹ recent stroke, frequent TIAs (>1/day), chronic renal insufficiency (creatinine greater than 1.2), increasing age, increasing volume of contrast medium, and longer procedure time were associated with higher complication rates. Davies ²⁵⁸ found that the rate of complications was not associated with the presenting symptoms (TIA, amaurosis, or cardiovascular accident [CVA]), though Hankey ²⁵⁹ reported an increased rate of periangiographic complications associated with CVA as the indication. Both Davies ²⁵⁸ and Hankey ²⁵⁹ found increasing degree of carotid stenosis associated with increased rates of complications.

Summary

In our systematic review of prospective studies examining the incidence of stroke and death following cerebral angiography in patients suspected of having cerebrovascular disease and potential candidates for CEA, the overall rate of 0.02 percent for deaths was lower than the 0.08 percent rate previously reported. ²⁴⁸ Only two deaths were found in 10 studies including 3,074 patients.

We found significant heterogeneity between rates of combined stroke or death from all studies as well as studies stratified by various methodological criteria. The rate of combined stroke or death ranged from 0 percent to 4 percent in three studies rated as having good quality, with the study rated as highest quality ²⁵⁹ reporting a rate of 1.3 percent (95 percent CI, 0.5 to 2.8 percent).

The risk of complications appears higher in patients with greater degrees of carotid stenosis, who are also those most likely to benefit from subsequent CEA. The magnitude of incremental risk of cerebral angiography (i.e., the risk above the baseline risk of recurrent stroke or death in recently symptomatic patients) cannot be reliably estimated at this time but would be expected to be lower than the rates reported above. ²⁷²

Study Characteristics and Findings, Randomized Controlled Trials

We identified four randomized controlled trials (RCTs) published since 1980 evaluating the efficacy of "best medical treatment plus carotid endarterectomy" versus "best medical treatment" alone for the treatment of symptomatic carotid artery stenosis.^{229, 230, 273-276} In one study, the trial was aborted after only 41 patients had been entered because a high rate of early complications was observed in the surgical arm. ²⁷³ Because CEA was routinely performed with a femoro-carotid shunt, a technique now rarely used, this study was excluded from further analysis.

Of the three other RCTs, two were large multicenter trials^{229, 230, 275, 276} and one a small multicenter trial. ²⁷⁴ In the VA Cooperative Study (VACSP), 193 recently symptomatic patients with completed stroke or transient ischemic attack and ipsilateral stenosis greater than 50 percent were randomized. ²⁷⁴ In NASCET and ECST, a total of 5,950 recently symptomatic patients with non-disabling stroke or transient ischemic attack and some degree of ipsilateral stenosis were entered, with approximately equal numbers of subjects in the two studies.^{230, 275-277} In all three trials, randomization occurred after cerebral angiography; complications due to cerebral angiography were not reported. Participants were randomized to "best medical treatment" alone, generally an antiplatelet agent and risk factor modification, versus "best medical treatment plus carotid endarterectomy," with surgical technique and timing generally left to the discretion of the participating surgeon.

All three trials were of good quality using USPSTF criteria for grading RCTs, with the following considerations: surgeons and patients could not be blinded to treatment, and outcome assessment did not appear to be blinded. In addition, NASCET screened participating centers for minimum volume of procedures (more than 50 CEAs over the preceding 24 months) and low pre-trial morbidity and mortality (less than 6 percent combined perioperative stroke and death), raising concerns about generalizability to other surgical settings. In the trials, the majority of participants were men (around 70 percent) and relatively young (mean age around 63 to 65 years), with low representation of non-whites (less than 10 percent). Although all participants were considered fit for surgery prior to entry, cardiovascular risk factors typically seen in patients with cerebrovascular disease were well represented. Mean followup was nearly 1 year in VACSP, 2 to 5 years in NASCET, and 6.1 years in ECST. Although NASCET and ECST used different methods to calculate the degree of angiographic carotid stenosis, the estimate of stenosis can be converted from one method to the other using validated formulas.^{44, 45}

In each of the three major trials (ECST, NASCET, VACSP), CEA appeared beneficial in patients with at least moderately severe stenosis. In VACSP, patients with >50 percent stenosis experienced a 1-year absolute risk reduction of 11.7 percent (7.7 percent surgical arm vs. 19.4 percent medical arm) for any stroke or crescendo TIA. ²⁷⁴ In ECST, there was a 3-year absolute risk reduction of 11.6 percent (14.9 percent surgical arm vs. 26.5 percent medical arm) for the endpoints stroke or death when the stenosis was 80 to 99 percent (equivalent to 60 to 99 percent by NASCET criteria). ²³⁰ In NASCET, for patients with 70 to 99 percent stenosis, there was a 2-year absolute risk reduction of 17 percent (9 percent surgical arm vs. 26 percent medical arm) for any ipsilateral stroke; for patients with 50 to 69 percent stenosis, there was a 5-year absolute risk reduction of 6.5 percent (15.7 percent surgical arm vs. 22.2 percent medical arm) for the same outcome.^{275, 277} In both NASCET and ECST, there was either no benefit or a trend toward harm in patients with lesser degrees of stenosis.^{230, 275, 277}

Study Characteristics and Findings of Systematic Review

A recent systematic review ²⁷⁸ examined the methodology and results of RCTs of CEA in symptomatic patients. The systematic review identified the same four RCTs described above, and also excluded the study by Shaw. In addition, the systematic review excluded VACSP in its summary measures because the result of "death or disabling CVA" (the major endpoint of the systematic review) could not be calculated from the reported results. ²⁷⁴ Because of generally similar results for less serious outcomes (stroke and crescendo TIA), smaller number of patients and events (particularly deaths), and shorter duration of followup compared to NASCET and ECST, including the results of VACSP would likely not alter the findings of the systematic review. We rated this systematic review as being of good quality, finding no substantial concerns regarding methodology. We did not identify any RCT comparing CEA with medical treatment alone published since the systematic review.

Using pooled data from NASCET and ECST, the systematic review found that CEA plus best medical treatment was effective in reducing the risk of disabling stroke or death compared to medical treatment alone, in patients with symptomatic carotid stenosis greater than 50 percent by NASCET criteria (greater than 70 percent by ECST method). ²⁷⁸ The degree of benefit increased with greater severity of stenosis. There was no significant heterogeneity between studies.

For patients with severe stenosis (greater than 80 percent by ECST or 70 percent by NASCET method), surgery reduced the relative risk of disabling stroke or death by 48 percent (95 percent CI, 27 to 73 percent). For patients with moderate stenosis (ECST 70 to 79 percent or NASCET 50 to 69 percent), surgery reduced the relative risk by 27 percent (95 percent CI, 15 to 44 percent). The number needed to treat to prevent one disabling stroke or death over 2 to 6 years was 15 (95 percent CI, 10 to 31) for severe stenosis and 21 (95 percent CI, 11 to 125) for moderate stenosis. For patients with lesser degrees of carotid stenosis (less than 70 percent by ECST or 50 percent by NASCET method), surgery increased the risk of disabling stroke or death by 20 percent (95 percent CI, 0 to 44 percent), with a number needed to harm of 45 (95 percent CI, 22 to infinity).

In both ECST and NASCET, multivariate analysis was performed to determine clinical and demographic factors associated with increased benefit.^{229, 230, 275} In ECST, increased age was associated with improved outcomes by a complex function; women had significantly less benefit from surgery than men. ²³⁰ In NASCET, for subjects with 70 to 99 percent stenosis, advanced age (over 70 years old) and male gender were associated with increased benefit. For patients in NASCET with 50 to 69 percent stenosis, age was not associated with increased benefit. In this subgroup, male gender was associated with increased benefit, with the number needed to treat to prevent one disabling stroke or death being 16 for men compared to 125 for women. No significant benefit was seen in the subgroup of women in NASCET with 50 to 69 percent stenosis; endarterectomy reduced the baseline risk of stroke from 15 percent to 14 percent, compared with a reduction of 25 percent to 17 percent in men.

In the subgroup of patients with 70 to 99 percent (i.e., severe) stenosis, the degree of benefit was related to the severity of stenosis. In NASCET, for stenosis 70 to 79, 80 to 89, and 90 to 99 percent, respective absolute risk reductions were 12, 18, and 27 percent, with corresponding numbers needed to treat 8, 5, and 4. ²⁷⁵

Although both ECST and NASCET evaluated the relationship between race and outcomes by multivariate analysis and found no significant association, non-whites were underrepresented in these studies.^{230, 275-277}

It must be noted that among patients screened in the NASCET, fewer than one third were randomized. ²⁷⁹ Approximately one third did not fulfill baseline criteria, 15 percent were excluded for medical reasons, and another 23 percent were eligible but not randomized. Such exclusions must be considered when trying to generalize data from the endarterectomy trials to individual patients or populations of patients in "real-world" health care settings.

Summary

In two large, good-quality RCTs, carotid endarterectomy reduced the risk of disabling stroke or death for surgically fit patients with symptomatic ipsilateral stenosis greater than 70 percent as measured by ECST, and over 50 percent as measured by NASCET. In a meta-analysis of these trials, the number needed to treat to prevent one disabling stroke or death over 2 to 6 years was 15 (95 percent CI, 10 to 31) for severe stenosis (70 to 99 percent by NASCET criteria or 80 to 99 percent by ECST criteria) and 21 (95 percent CI, 11 to 125) for moderate stenosis (50 to 69 percent NASCET or 70 to 79 percent ECST). No benefit was seen in patients with lesser degrees of carotid stenosis. In the subgroup of patients with severe stenosis, increased degree of stenosis was associated with greater benefit from surgery. The results of the studies are generalizable to surgeons and centers with low perioperative complication rates (30-day stroke or death rate less than 6 percent). The studies did not include angiographic morbidity or mortality in their results.

Although patients over 80 years old, non-whites, and females were underrepresented in these studies, multivariate analysis to determine factors associated with increased benefit was performed on these and other clinical and demographic characteristics in the two RCTs. In NASCET and ECST, less benefit was seen in females for all degrees of carotid stenosis, and in women with 50 to 69 percent stenosis, the absolute risk reduction was eight-fold lower in women than in men. The lesser degree of benefit for women may be partially due to a lower baseline recurrent stroke rate compared to men for equivalent degrees of carotid stenosis. ²²⁹ Older age was associated with increased benefit in ECST and in the subgroup of patients in NASCET with 70 to 99 percent stenosis.^{230, 275, 276}

Study Characteristics and Findings, Systematic Review

We identified one systematic review ²⁸⁰ that examined the perioperative (30-day) complication rate of CEA in symptomatic patients in studies published since 1980. This systemic review found a total of 51 studies in which the complication rate in symptomatic patients could be calculated separately from the rate in asymptomatic patients.

The systematic review did not rate the quality of the included papers, and included all studies irrespective of methodology. No other major methodological areas of concern were identified using USPSTF criteria for systematic reviews, and the study was given an overall fair quality rating. The summary outcomes were perioperative (30-day) deaths, and perioperative combined deaths or strokes. The study evaluated the effect of the following methodological features on reported complication rates: the use of prospective or retrospective data, author departmental affiliations, independent or non-independent assessment of outcomes, and year of publication. ²⁸⁰ Author departmental affiliation and independence of ascertainment were assessed together in four categories (see below).

The systematic review found significant heterogeneity in the reported rates of perioperative complications (chi-square=203, df=49, P<.001). The overall mortality due to CEA for symptomatic stenosis was 1.6 percent (95 percent CI, 1.3 to 1.9, n=17,105); the overall risk of stroke or death was 5.64 percent (95 percent CI, 4.4 to 6.9, n=15,956). Single surgeon author/non-independent assessment studies were associated with the lowest rate of complications (2.3 percent stroke and/or death), followed by multiple surgeon authors/non-independent assessment (5.5 percent), non-surgeon authors/non-independent assessment (6.4 percent), and any author/independent assessment studies (7.7 percent). ²⁸⁰ Although there was a trend toward higher complication rates in prospective studies, this was not significant. Including studies published prior to 1980 (from a prior study by the same author), no difference in complication rates according to year of publication was found (including studies published before 1980). If the analysis was limited to studies published since 1980, however, there appeared to be a trend toward higher reported complication rates in more recent studies. The difference in complication rates according to authorship/independent assessment of outcomes was significant when examined in multivariate analysis looking at other study characteristics (retrospective or prospective and year of publication) and was thought to explain much of the heterogeneity in reported rates. ²⁸⁰

Findings of Individual Studies

In order to assess the influence of methodological quality and other study characteristics not examined in the prior systematic review, ²⁸⁰ we performed a literature search and selective review of studies examining the rate of perioperative complications in patients with symptomatic carotid artery stenosis. We limited our review to studies in which the complication rate for symptomatic patients with carotid stenosis could be calculated separately from the rate in asymptomatic patients, because of evidence that symptomatic patients are at higher risk for perioperative complications. ²⁸¹ We identified studies published since 1980 that met our inclusion criteria.

We included all prospective studies in our review. We limited inclusion of retrospective studies to several defined subgroups, because on our initial review of abstracts we found large numbers of retrospective studies that generally appeared to be of poorer quality than the prospective studies. We included population-based retrospective studies because these appeared more likely to have independent assessment, multidisciplinary or non-surgeon authorship, and larger sample sizes than non-population-based retrospective studies. We defined "population-based" as studies that attempted to capture all or a clearly defined subset (e.g., Medicare-insured or a random sample) of patients from a specified geographic area. We also included retrospective single surgeon author studies in order to explore possible reasons for the very low complication rates found in the previous systematic review. ²⁸⁰ Because we also reviewed all studies on timing of CEA (see next section), we included these studies, assuming that they represented a non-biased sample of retrospective studies of varying methodological quality that were not population-based or single surgeon author studies. Finally, we included studies classified as prospective by Rothwell ²⁸⁰ but re-classified as retrospective after detailed review of study methods. We believed that these studies would provide another non-biased sample of retrospective studies that were not population-based or single surgeon author studies.

We reviewed all 19 studies classified as prospective in the previous systematic review, ²⁸⁰ substituting final results from ECST ²³⁰ for the earlier interim results ²⁷⁶ and Magee ²⁸² for Earnshaw ²⁸³ (a less detailed report on the same population). In eight of these studies^284-291 we found no clear evidence that the data had been collected prospectively, and we re-classified these studies as retrospective. Two of these "re-classified" studies^{284, 289} were found to be single surgeon author studies. One prospective study was population-based. ²⁹² We identified four prospective studies published since 1996^{277, 293-295} and four studies published prior to 1996 not included in the original systematic review,^296-299 resulting in a total of 18 included prospective, non-population-based studies and six "re-classified" retrospective, non-single surgeon author studies.

We identified a total of six single surgeon author studies. Five studies^{284, 300-303} were identified as such in the previous systematic review, ²⁸⁰ and one was a study previously classified as having multiple surgeon authors but found on review to have a single surgeon author. ²⁸⁹ We did not identify any prospective single surgeon author studies published since 1990. All single surgeon author studies were retrospective.

We identified a total of eight population-based studies. We reviewed the two population-based studies^{292, 304} previously identified by Rothwell, but used data from Brott ³⁰⁵ instead of Kempczinski ³⁰⁴ because of more detailed reporting of complications from the same population. We identified four additional population-based retrospective case series published since 1996^306-309 and two population-based studies published prior to 1996 not included in the systematic review.^{310, 311} One of the population-based studies ²⁹² was prospective.

We included nine retrospective studies^312-320 that examined the association between timing of CEA and perioperative complication rates. We excluded one by Gasecki, ³²¹ a post-hoc analysis of NASCET, because its results were reported in other included studies.^{275, 277}

In the studies that we reviewed, we did not distinguish between those routinely using a shunt during surgery, those that used or did not use particular patching procedures, or studies that used different anesthetic techniques. Systematic reviews on these subjects^322-325 have shown inadequate evidence to suggest differences in clinical outcomes between these techniques. We reviewed recent trials on primary closure versus patching ³²⁶ and the practice of eversion endarterectomy^{327, 328} and did not find sufficient evidence to suggest that the conclusions of the systematic reviews would be different with their inclusion.

The 18 prospective studies, six retrospective single surgeon author studies, eight population-based studies, and six "re-classified" retrospective studies (Evidence Table 12) were abstracted with regard to year of publication, setting, type of study (RCT, population-based), prospective or retrospective collection of data, author departmental affiliation, and major outcomes (stroke or death) in the perioperative (30-day) period. The retrospective studies on timing of CEA had previously been abstracted (Evidence Table 13). When studies included both symptomatic and asymptomatic patients, we abstracted rates of complications for symptomatic patients only. Overall quality of the studies was determined using an eight-criterion scoring system derived from USPSTF quality criteria for cohort studies, modified for studies reporting rates of complications (see Methods). Each quality criterion was abstracted for all reviewed studies. Adequate duration of followup was defined as 30 days, and independent ascertainment defined as non-surgeon ascertainment. We also compiled complication rates, author affiliation/independent assessment categorization, and year of publication for all non-included retrospective studies examined in the previous systematic review; ²⁸⁰ we did not further rate the quality of these papers (number of studies=23).

Twelve studies were rated as good quality. Of these, five studies (reporting results from three trials) were RCTs,^{229, 230, 274, 275, 299} six were population-based studies,^{305-307, 309-311} and one was a non-population-based observational study. ³²⁹ NASCET and ECST, the two largest RCTs, were found to have no major methodological areas of concern,^275-277 adequately meeting all quality ratings criteria for studies reporting complication rates (see Methods). In these studies, the perioperative complication rates ranged from 6.7 percent (73/1,087) ²²⁹ to 8.0 percent (19/327) ²⁷⁵ for stroke or death, and 0.6 percent (2/327) ²⁷⁵ to 1.3 percent (22/1,787) ²⁷⁶ for death alone. In the largest of the good-quality population-based studies, which also had no significant methodological areas of concern, the rate of CVA or death was 6.4 percent (60/943), and death alone 1.7 percent (16/943). ³⁰⁷ The single non-population-based study rated as good quality did not adequately define complications and had inadequate statistical analysis of potential confounders, but otherwise appeared to adequately meet all quality ratings criteria. ³²⁹ This study reported a 14.5 percent (16/110) rate of CVA or death, and a 3.6 percent (4/110) rate of death alone.

Other important perioperative complications (non-fatal cardiovascular or pulmonary complications) were not routinely assessed or reported, although a rate of 0.9 percent (3/327) was reported in NASCET. ²⁷⁵

We rated 21 studies as poor quality^{284, 286-291, 295, 298, 300-303, 312-314, 316, 318-320, 330} and 14 as fair quality.^{282, 285, 292-294, 296, 297, 308, 315, 317, 331-334} All six single surgeon author studies were classified as poor quality.^{284, 289, 300-303} In the studies rated as poor or fair quality, there was substantial variation in reported complication rates. Excluding studies with smaller (n<100) sample sizes, the rates of death ranged from 0.4 percent (1/274) ³¹² to 3.5 percent (15/427), ³³¹ and the rates of combined stroke or death ranged from 1.1 percent (3/274) ³¹² to 10.3 percent. ²⁸²

Pooled Findings

We did not pool rates of perioperative outcomes from all studies that we reviewed, because we found significant heterogeneity for the rates of the perioperative outcomes of death alone (chi-square=115, df 64, p<0.0001) in 65 studies and combined stroke or death (chi-square=338, df 68, p<0.00005) in 69 studies (outcomes available from 47 studies that we abstracted and 23 studies previously reviewed by Rothwell ²⁸⁰ ). In the studies we reviewed, we found much less heterogeneity between the results of studies rated good quality for both the outcomes of death alone (12 studies, chi-square=21.5, df 11, p=0.03) and combined stroke or death (nine studies, chi-square=13.8, df 8, P=0.09). Pooled event rates from these studies were 1.6 percent (95 percent CI, 1.0 to 2.5 percent) for death alone and 6.8 percent (95 percent CI, 4.6 to 9.5 percent) for combined stroke or death.

Risk Factors for Perioperative Complications

A systematic review of 14 potential risk factors for perioperative stroke or death from CEA combined data from 36 studies to calculate odds ratios using stepwise logistic regression analysis. ³³⁵ The systematic review did not rate the quality of included studies, and included rates for both symptomatic and asymptomatic patients. The review found that the odds of stroke and death were decreased in patients with ocular ischemia alone compared to those with cerebral TIA or stroke (OR 0.49). The odds were increased in women (OR 1.44); subjects aged over 75 years (OR 1.36); and patients with systolic blood pressure over 180 (OR 1.82), peripheral vascular disease (OR 2.19), occlusion of the contralateral internal carotid artery (OR 1.91), stenosis of the ipsilateral internal carotid siphon, and stenosis of the ipsilateral external carotid artery (OR 1.61). Operative risk was not significantly related to presentation with cerebral TIA vs. stroke, diabetes, angina, recent myocardial infarction, current cigarette smoking, or plaque surface irregularity at angiography.

Detailed multivariate analyses of potential risk factors for perioperative complications of CEA were performed in ECST, ²⁷⁶ and in a recent good-quality population-based study of Medicare patients. ³⁰⁶ The latter found that surgery at a higher-volume hospital was associated with a 71 percent reduction in risk of stroke or death at 30 days after CEA. Other factors associated with increased risk of perioperative complications in multivariate analysis were TIA as indication (OR 2.9), history of angina (OR 2.4), and the presence of renal insufficiency (OR 3.3). Age and gender were not independent predictors of perioperative risk. In ECST, female gender (HR 2.39) and age (HR 0.959/years at randomization) were associated with increased risk 0 to 5 days after surgery.

Supplemental Analyses

We performed several analyses to assess the relative influence of study characteristics on reported complication rates. Because the pooled absolute rate of death did not vary much when studies were stratified according to various study characteristics (range 1.1 to 2.1 percent), we report the stratified results only for the combined endpoint of stroke or death, which showed more substantial variation (range 2.3 to 9.0 percent).

In order to determine whether the conclusions of the previous systematic review ²⁸⁰ remain robust after the inclusion of additional studies and re-classified data from studies previously analyzed, we stratified the studies that we abstracted by year of publication, retrospective or prospective collection of data, and by author departmental affiliation. For these analyses, we included the results of studies that we had reviewed (n=46) as well as those studies reported in the previous meta-analysis (n=23). ²⁸⁰ In contrast to the findings of Rothwell, ²⁸⁰ there was a significantly lower pooled rate of combined stroke and death in retrospective studies (4.5 percent; 95 percent CI, 3.8 to 5.3 percent) compared to prospective studies (7.0 percent; 95 percent CI, 5.3 to 9.0 percent). Like Rothwell, we found a non-significant trend toward a higher pooled rate of complications in more recent (published 1990 or after) studies (5.5 percent; 95 percent CI, 4.2 to 6.9 percent) compared to earlier (1980 to 1989) studies (4.8 percent; 95 percent CI, 3.9 to 5.8 percent). We also found that complication rates were significantly different according to author departmental affiliation: 2.3 percent (95 percent CI, 1.4 to 3.4 percent) for single surgeon author studies, 4.5 percent (95 percent CI, 3.8 to 5.4 percent) for multiple surgeon author studies, and 6.6 percent (95 percent CI, 5.4 to 7.8) for studies with at least one non-surgeon author. Finally, like the previous meta-analysis, ²⁸⁰ we found a higher rate of complications in studies with independent ascertainment (7.6 percent; 95 percent CI, 5.9 to 9.6 percent) compared to those with non-independent ascertainment (4.3 percent; 95 percent CI, 3.6 to 5.0 percent).

In order to evaluate the effect of different study designs on reported complication rates, we assessed two different study design types and their association with complication rates. For these analyses, we included only those studies in which we reviewed the study methods (n=46). First, to assess whether RCTs truly lack generalizability (i.e., report lower complication rates) because of stringent selection of patients, surgeons, and hospitals, we stratified studies as RCTs or non-RCTs. Next, we stratified studies as population-based or non-population-based to examine whether population-based studies, which were rated as generally higher quality, are associated with higher rates than non-population-based studies (Table 15).

Table

Table 15. Perioperative (30-day) complication rates of carotid endarterectomy, stratified by study characteristics.

RCTs had a perioperative complication rate of 9.0 percent (95 percent CI, 5.6 to 13.8 percent), and non-RCTs 4.4 percent (95 percent CI, 3.9 to 6.0 percent). Higher complication rates in the RCTs may have been due to better methodological techniques for identifying complications. Population-based studies had a rate of 6.4 percent (95 percent CI, 4.0 to 9.5 percent), and non-population-based studies 5.2 percent (95 percent CI, 4.0 to 6.5 percent).

The association between higher rates of perioperative complications for RCTs, population-based studies, non-surgeon author studies, and prospective studies may be suggestive of an association between assessed study quality and reported complication rates, as these studies all have in common higher average quality.

Validation of Quality Ratings Criteria

To validate the quality ratings criteria that we developed to assess studies reporting complication rates and to further explore the clustering of higher complication rates that we observed in our initial supplemental analyses, we performed analyses on each of the eight quality ratings criteria, as well as on the overall quality rating (poor, fair, or good) derived from these criteria.

Complication rates correlated with overall quality rating, with poor-quality studies reporting a rate of stroke or death of 3.8 percent (95 percent CI, 2.7 to 5.2 percent), fair quality studies 6.4 percent (95 percent CI, 4.5 to 8.7 percent), and good quality studies 6.8 percent (95 percent CI, 4.6 to 9.5 percent) (Table 15).For all individual quality ratings criteria that we could assess (perioperative followup was reported as complete in all studies), meeting the criterion adequately was associated with higher reported complication rates.

Summary

Using data from 12 studies of good quality, the pooled rate of combined perioperative (30-day) stroke or death was 6.8 percent (95 percent CI, 4.6 to 9.5 percent), and from nine studies of good quality, the pooled rate of death alone was 1.6 percent (95 percent CI, 1.0 to 2.5 percent).

In the NASCET trial, the 30-day post-randomization rate of death or stroke ranged from 2.4 percent (for patients with <70 percent carotid stenosis) to 3.3 percent (70 to 99 percent stenosis) in patients assigned to medical therapy.^{275, 277} Therefore, surgery is associated with an additional 35 to 44 perioperative events per 1,000 patients. In NASCET, approximately 60 percent of the strokes that occurred in the perioperative period were non-disabling (Rankin score <3).

There was significant variation in complication rates across studies, which was partly explained by methodological characteristics of the studies. Population-based studies, randomized controlled trials, studies with independent ascertainment of complications, studies with non-surgeon authors, and studies published since 1990 were associated with higher combined complication rates. The pooled complication rate in RCTs was higher than the pooled rate for non-RCTs, suggesting that RCTs may have high generalizability despite strict selection criteria. Population-based studies also reported relatively high perioperative complication rates. All of the characteristics associated with higher complication rates appear to occur in studies rated as having higher average methodological quality.

Background

The timing of carotid imaging in patients with symptomatic cerebrovascular disease has important implications for cost and effectiveness in everyday practice. Published guidelines recommend prompt evaluation of TIAs with carotid imaging (within 1 week), and in some clinically higher-risk patients, hospitalization to facilitate workup. ³³⁶ Advocates of early testing argue that if the study is not done during the hospitalization or otherwise expedited, there is a risk that it may never be done because of poor communication between inpatient and outpatient providers, lack of available facilities in some settings, or because the patient fails to seek followup. Keeping patients in the hospital for diagnostic testing, admitting patients with symptoms for testing, and maintaining readily accessible and reliable outpatient carotid imaging services is costly, however, and may not improve rates of diagnostic testing. We did not identify any studies reporting the rates of carotid imaging in patients with recently symptomatic cerebrovascular disease who did not receive "expedited" carotid imaging.

A second argument for early diagnostic testing is that early CEA might be beneficial. The safety of early CEA, however, remains controversial. Early experience with CEA suggested a high rate of complications (in particular ICH) when performed early for symptomatic carotid stenosis, compared with delayed or elective surgery. ³³⁷ A recommended waiting period of 4 to 6 weeks evolved from these early observations. More recently, however, some surgeons have questioned the necessity of waiting for 4 to 6 weeks because of clinical observations and published series of early CEA without excess morbidity or mortality. Proponents of early CEA have argued that the time immediately following symptoms may be a high-risk period for recurrent cerebrovascular events, and delays in performing CEA could attenuate the beneficial effects of the procedure. In NASCET, for example, 2.4 percent of medically treated patients with less than 70 percent stenosis and 3.3 percent with 70 to 99 percent stenosis had recurrent stroke or death within 30 days of entering the trial.^{229, 275} Uncertainty also remains regarding the existence and magnitude of excess risk of early CEA in patients with different categories of ischemic brain injury (TIA, CVA with normal or abnormal CT scan). If early CEA is not associated with excess morbidity or mortality, it may be important to obtain diagnostic imaging (i.e., CUS, MRA, or angiography) early after presentation with cerebrovascular symptoms.

Findings

We identified 10 papers with original data that specifically evaluated the timing of CEA and met our inclusion criteria.^312-321 All of these papers examined the perioperative complications of CEA performed less than 4 to 6 weeks after presenting with cerebrovascular symptoms, versus later CEA. We identified one non-systematic review, ³³⁸ but no systematic reviews (Evidence Table 13).

The definition of "early" CEA varied between studies, ranging from less than 2 weeks to less than 6 weeks after presenting with symptoms. In three studies,^{312, 318, 319} patients undergoing early CEA had the procedure within 2 to 3 weeks after becoming clinically stable. No paper specifically addressed the issue of increased risk of performing CEA in a more urgent manner, i.e., within several days to a week of presenting with symptoms. The degree of risk of very early CEA could have a greater impact on the need to perform urgent diagnostic testing of the carotid arteries. Although Paty ³¹⁶ reported the rate of complications of CEA performed within one week of symptoms, only 31 procedures were performed in this subgroup.

We excluded one study that reported early CEA in a series of three subjects. ³³⁹ Another paper ³²⁰ was an update of a previously published report, ³⁴⁰ and we used data from the later study. Of the 10 included studies, one was a post-hoc subgroup analysis of patients randomized to CEA (NASCET), ³²¹ and the remainder were retrospective reports of patients undergoing early CEA at one or two centers.^{312-314, 318-321} We did not identify any randomized trial specifically designed to address safety of early CEA. Ad-hoc cohorts (i.e., no true inception cohort) of patients undergoing early CEA versus later CEA could be analyzed for all studies except one ³¹³ in which there were no data regarding patients undergoing later CEA.

The quality of the papers was rated as fair (three studies^{315, 317, 321}) or poor (seven studies^{312-314, 316, 318-320}) using USPSTF criteria for cohort studies in conjunction with criteria adapted for studies reporting complication rates (Evidence Table 13) (see Methods); no paper was rated as being of good quality. We defined adequate length of followup as 30 days (by convention), and independent ascertainment as being performed by a non-surgeon. Important methodological problems included the uniform lack of true inception cohorts, lack of statistical analysis of potential confounders, poorly defined complications, unclear ascertainment methods, unclear selection methods for early surgery, and unclear duration of followup. In some studies, the ad-hoc cohorts of early and later CEA had differences in important clinical characteristics that made it likely that they were not comparable in terms of perioperative risk.^{312, 318} For example, in one study, ³¹² patients were selected for earlier CEA based on having a normal CT scan. In other studies, there was inadequate information given to compare clinical characteristics of the patients undergoing early and later CEA.^{313, 314, 316, 318-320} There were also concerns about potential publication bias, as surgeons with low complication rates with early CEA may have been more likely to publish their results. Consistently reported outcomes were death and stroke, typically not classified by severity.

Of the three studies rated as having overall fair quality, one, ³²¹ a post-hoc analysis from the NASCET, was rated as having the highest methodological quality. This study included prospective entry of subjects, independent ascertainment of events, and comparable clinical and demographic characteristics between early and later CEA groups. Even in this study, however, methods of selection for early surgery were unclear, and no statistical analysis of potential confounders or risk factors was performed. In this study, the rate of perioperative stroke was 4.8 percent (2/42) for early CEA vs. 5.2 percent (3/58) for later CEA (difference not significant); no deaths were reported in either group. ³²¹

In the other included studies, there was no clear pattern of increased complications in either the early or later CEA groups. Some studies^{314, 316, 320} reported an increased rate of complications in the early CEA group; others^{312, 315, 317} reported an increased rate in the later CEA group. In general, the numbers of events and differences in rates between groups was too small to be statistically significant, except for one study ³¹⁴ that found a statistically significant difference between a rate of stroke or death of 18.5 percent (5/27) for early CEA vs. 0 percent (0/22) for later CEA. In the one study reporting complication rates in patients undergoing very early CEA (<1 week after the initial cerebral ischemic event), the rate of perioperative stroke was 3.2 percent (1/31); perioperative deaths were not reported according to timing of surgery. ³¹⁶

Although the studies in general included stable patients without nondisabling stroke, interpretation of the findings is complicated by differences in the specific inclusion criteria in each study. For example, one study ³²¹ included patients with 70 to 99 percent stenosis with or without abnormal CT scans; another ³¹² studied only those with normal CT scans. Other studies did not adequately define patient characteristics, making it difficult to generalize their results, as the risk of perioperative complications may vary according to the presenting cerebrovascular symptoms. In those studies^{312, 318, 321} reporting outcomes according to findings on CT scan, the number of events was too low and the difference in complication rates too small to detect significant differences in perioperative complications for early CEA according to CT scan findings. Results were also not reported for important clinical subgroups, including those with greater degrees of symptomatic carotid stenosis.

Pooled Findings

We did not find significant heterogeneity between rates of combined stroke or death from all studies (chi-square=11.12, df 8, p=0.19) or from the three studies rated as having fair quality (chi-square=1.30, df 2, p=0.52). In the fair-quality studies, the pooled rate of all perioperative complications (stroke or death) in patients undergoing early CEA was 3.3 percent (6/210; 95 percent CI, 0.3 to 12.0 percent), compared to 5.3 percent (12/226; 95 percent CI, 2.0 to 10.2 percent) in those undergoing later CEA (Evidence Table 13). This difference was not statistically significant. For the fair-quality studies, the pooled rate of death was 1.5 percent (2/210; 95 percent CI, 0.08 to 6.0 percent) in patients undergoing early CEA versus 4.8 percent (4/226; 95 percent CI, 0.0012 to 58 percent) in those undergoing later CEA; this difference also was not statistically significant.

Using data from all included studies, the pooled rate of all perioperative complications (stroke or death) in patients undergoing early CEA was 3.9 percent (25/685; 95 percent CI, 1.8 to 6.7 percent) compared to 2.7 percent (31/1,130; 95 percent CI, 1.2 to 4.8 percent) in those undergoing later CEA (Evidence Table 13). This difference was not statistically significant. An increased pooled rate of death alone (1.0 percent [5/478] for early CEA vs. 1.0 percent [11/973] for later CEA) was not observed.

The overall (later and early CEA combined) complication rates (death 1.1 percent [16/1451], stroke or death 3.1 percent [56/1815]) from these studies are comparable to the CEA complication rates reported in other recent studies of comparable quality which did not specifically address the issue of timing.

Supplemental Analyses

We stratified results by two study characteristics. First, in order to examine the potential impact of improvements in early CEA technique or better selection methods for early CEA, we stratified results by publication year (prior to 1990 vs. after 1990) to determine whether the risk of early CEA has decreased over time. Second, in order to examine the possibility of publication bias, we stratified results according to author departmental affiliation (only surgeon authors vs. non-surgeon authors) to determine whether studies by surgeon authors were more likely to report favorable results for early CEA (Evidence Table 13). We did not find significant heterogeneity for the results of the studies stratified into each of these subgroups. Because all of the studies were ad-hoc cohorts, we did not stratify by study type. There were insufficient data to stratify studies by severity of symptoms, degree of carotid stenosis, or CT scan findings.

In studies with non-surgeon authors, early CEA was not associated with a higher pooled risk of adverse events (early CEA 4.2 percent [95 percent CI, 0.5 to 14.3 percent]; later CEA 5.2 percent [95 percent CI, 2.3 to 9.8 percent]). In studies with only surgeon authors, in contrast, the pooled risk of early CEA (4.0 percent; 95 percent CI, 1.2 to 8.9 percent) was higher than the pooled risk of later CEA (1.9 percent; 95 percent CI, 0.8 to 3.3 percent), though this result was not significant. Publication bias could explain this trend if surgeon authors were more likely to publish reports showing poorer outcomes of early CEA.

In studies published prior to 1990, early CEA was associated with higher pooled complication rates (5.8 percent; 95 percent CI, 1.3 to 14.8 percent) than later CEA (2.4 percent; 95 percent CI, 0.9 to 4.6 percent). In contrast, in studies published in 1990 or later, the pooled rate of complications was lower for early CEA (2.7 percent; 95 percent CI, 0.5 to 7.3 percent) than for later CEA (4.0 percent; 95 percent CI, 1.3 to 7.4 percent). Neither of these differences was statistically significant. The pooled rate from pre-1990 studies is not significantly different from the pooled rate of post-1990 studies (chi-square=1.31, df 1, p=0.25), though the results suggest a trend toward improved selection methods for early CEA or better surgical techniques over time.

Summary

There is fair evidence that early CEA is not associated with an increased risk of major complications. Three non-randomized studies of fair quality suggest that in patients with recent minor or non-disabling stroke, CEA performed earlier than the traditional waiting period of 4 to 6 weeks is not associated with significantly increased adverse events compared to delayed surgery, with a pooled rate of 3.3 percent for early CEA versus 5.3 percent for later CEA. When data from all studies (including seven rated poor quality) are included, the rate of pooled complications is 3.9 percent for early CEA versus 2.7 percent for later CEA. The pooled rate of death alone from all studies was about 1.0 percent in patients undergoing either early or later CEA. There was a non-significant trend towards better outcomes for early CEA in studies published since 1990.

There is insufficient evidence to draw conclusions regarding the risk of very early CEA (i.e., less than 1 week after presenting with symptoms). There is also inadequate evidence to draw conclusions for specific subgroups, including patients with specific CT scan findings and greater degrees of carotid stenosis. Patients selected for early CEA in these studies are likely to comprise an overall lower-risk population compared to patients not selected for early CEA, though in higher-quality studies patients undergoing early and later CEA were comparable according to important clinical and demographic criteria.

Carotid Imaging Decision Model

The carotid imaging decision model follows a hypothetical cohort of patients with newly diagnosed, anterior circulation (carotid territory), ischemic (non-hemorrhagic) stroke over the course of their remaining lifetimes. The base case represents 65-year-old men with newly diagnosed ischemic stroke who have survived acute stroke treatment and are now at the point where a decision will be made about further diagnostic testing. They are assumed to have neither a prior indication for nor contraindication to CEA. The cohort is comprised of patients with 0 to 49 percent stenosis, 50 to 69 percent stenosis, 70 to 99 percent stenosis, or 100 percent occlusion. The model excludes the clinically unacceptable course of not providing at least standard medical treatment to all stroke patients. Its time horizon is 30 years, at which time virtually all patients will have died of stroke-related or other causes. Decisions to initiate or change therapy are based on clinical events alone.

Carotid Testing Strategies

The various testing strategies are listed in Table 16. The objective of testing is to accurately identify those stroke patients who are most likely to benefit from CEA. The baseline strategy is to provide standard medical treatment, including antiplatelet therapy (aspirin), to everyone (strategy 1). Alternate strategies include testing everyone using cerebral angiography (strategies 2 & 3), CUS (strategies 4 & 5), or MRA (strategies 6 & 7), with positives receiving a CEA. Strategies 8 to 11 are sequential and involve testing everyone with either CUS or MRA, then following up with cerebral angiography when the noninvasive test is positive. Finally, strategies 12 to 13 involve performing CUS and MRA jointly on everyone, reserving cerebral angiography for cases where CUS and MRA results differ. For each primary testing strategy, we examine the effect of including as "test positive" (and therefore referring for CEA) those with 70 to 99 percent stenosis (severe), and those with 50 to 99 percent stenosis, which includes moderate stenosis as well (Figure 8).

Table

Table 16. Carotid imaging model testing strategies.

Figure

Figure 8. Carotid Imaging decision tree. Numbers in parentheses refer to Table 16. *SMT -- standard medical treatment CEA -- carotid endarterectomy

Markov Modeling

The Markov nodes in the carotid imaging decision model are similar to those used in the echocardiography decision model and include five stroke-related health states: TIA, minor ischemic stroke, moderate ischemic stroke, severe ischemic stroke, and death (Figure 9). Transitions between states are as described for the echocardiography model. CUS and MRA are assumed not to be associated with significant complications. Cerebral angiography has a complication risk ranging from transient neurologic events to permanent neurologic deficits and death. Persons undergoing CEA assume a risk of periprocedural MI, stroke, or death. Persons who experience a MI undergo a permanent decrement in utility due to the MI.

Figure

Figure 9. Markov diagram--Post-treatment health states (carotid model).

The Markov nodes in the carotid imaging model are run up to 360 monthly cycles. As with the echocardiography model, we use life tables to establish the age- and gender-specific baseline mortality rate for the cohort and adjust these rates to account for the increased risk of cardiac-related deaths among patients with stroke. These rates are assumed to be highest for those with severe carotid stenosis or occlusion, intermediate for those with moderate stenosis, and lowest for those with mild or no stenosis. The carotid imaging model includes nine separate Markov nodes that reflect the various prognoses of person with four different degrees of stenosis (mild, moderate, severe, occluded), receiving standard medical treatment alone or along with CEA, and those experiencing complications of angiography, who are assumed not to undergo CEA.

Costs

As in the echocardiography model, we take the perspective of direct medical costs related to stroke evaluation and management, and exclude indirect costs related to lost work productivity. Cost estimates were derived from best estimates from the literature or from Medicare fee schedules (Table 17). All financial outcomes are adjusted to 2000 dollars using the medical component of the Consumer Price Index. For the purposes of calculation, the model assumes that events occur in the middle of each year.

Table

Table 17. Parameter List -- Carotid Imaging.

Major Assumptions and Estimates

Wherever possible, we used estimates in the cost-effectiveness model that were derived from our systematic review. For parameters not identified in our evidence review and for those for which our review revealed insufficient evidence, we either used the best available estimates from the literature or made informed assumptions, guided by our technical expert group, and tested the effects of those assumptions in sensitivity analyses. Major assumptions in our base-case analysis are listed in Table 17. Other assumptions included: the prognosis of a patient with occlusion is the same as that of a patient with severe (70 to 99 percent) stenosis receiving standard medical treatment; the duration during which the rate of recurrent stroke among patients with moderate or severe stenosis is lower for those receiving as compared to those not receiving CEA is four years, and after four years, the rate of recurrent stroke is similar between the two groups, such that their stroke-free survival curves run parallel; the severity distribution of non-fatal strokes is the same as that for initial strokes, i.e., 24 percent minor, 25 percent moderate, and 51 percent severe; the annual excess cardiovascular mortality among patients with stroke varies by degree of stenosis but does not vary with stroke severity; and persons with more severe strokes, and therefore worse health states, have a higher rate of death not attributable to cardiac causes or recurrent stroke.

Table

Table 18. Baseline carotid imaging results.

Results

Table 18 describes the baseline results of the carotid imaging decision model, which follows a cohort of 65-year-old white males. The 12 testing strategies are compared with the strategy of treating all patients nonsurgically (i.e., standard medical therapy alone). After removing both strongly and weakly dominated strategies, two strategies remain undominated: MRA with direct referral to CEA of patients with severe (70-99 percent) stenosis, and joint CUS and MRA, with direct referral to CEA of patients for whom both tests demonstrate moderate to severe (50-99 percent) stenosis, and angiographic confirmation when the two tests disagree. The incremental cost-effectiveness ratios for these two strategies are approximately $250,000 and $700,000 per QALY, respectively.

Table

Table 19. Sensitivity analyses--Carotid imaging.

Influence of Demographics

As with the echocardiography model, we used data from the U.S. Life Tables produced by the National Center for Health Statistics to adjust the baseline all-causes mortality rate in the model, which is for 65-year-old white males. We used the Life Tables to adjust baseline mortality according to various combinations of gender, race (specifically, African American), and age (55, 65, 75, and 85 years old). Figure 10 illustrates the effect of increasing baseline mortality (measured by declining average life expectancy) on the cost per QALY for the strategy of testing with MRA and directly referring to CEA those patients with severe stenosis. Readers can apply this information to the specific circumstances of their patients.

Figure

Figure 10. Effect of Baseline Life Expectancy on Cost-Effectiveness of MRA with Direct Referral of Patients with Severe (70-99 percent) Stenosis for CEA (MRA-70 percent).

Sensitivity Analysis

We performed sensitivity analyses on the model parameters listed in Table 17. Table 19 presents selected results. Varying the cost of testing did not substantively affect the results, except in the case where MRA was assumed to cost $2,500. In this analysis, the strategy of initial CUS with angiographic confirmation of severe stenosis became undominated, with a cost-effectiveness ratio of $280,000 per QALY. Varying the accuracy of the different testing strategies over wide ranges did not have a substantial overall effect on the results. When the perioperative complication rate was assumed to be zero, noninvasive strategies involving direct referral to CEA of patients with moderate or greater stenosis expectedly became the most cost-effective; without risk of complications, angiographic confirmation to avoid false positives was no longer beneficial, and the marginal benefit of CEA among patients with moderate stenosis was no longer counterbalanced by perioperative risk. Varying the duration of risk reduction associated with CEA between 2 and 10 years also did not substantively affect the cost-effectiveness ratios. Likewise, restricting the cohort to only patients with TIA or minor stroke, which reflects the patient populations in the two large carotid endarterectomy trials, did not have a major impact on cost-effectiveness ratios, though it did produce a different set of undominated strategies.

The variable with the greatest influence on the results of the carotid imaging model was the prevalence of severe carotid stenosis. At severe stenosis prevalences of 0.15 and below, all testing strategies were dominated by the strategy of no testing or had cost-effectiveness ratios exceeding $250,000 per QALY. However, as this prevalence increased above 0.15, the cost-effectiveness ratios of two strategies -- CUS with angiographic confirmation of severe stenosis (CUS/Angio-70), and MRA with direct CEA referral for severe stenosis (MRA-70) -- fell precipitously, such that at a prevalence of 0.20, these strategies had cost-effectiveness ratios in the range of $60,000 to $75,000 per QALY (Figure 11). At higher prevalences, these ratios fell further. When compared to the strategy of no testing, CUS/Angio-70 had an incremental cost-effectiveness ratio of less than $50,000 per QALY at a prevalence of 0.25, while the incremental cost-effectiveness of MRA-70 fell below $50,000 per QALY as the prevalence of severe stenosis approached 0.30. These results suggest that carotid imaging may compare unfavorably, in terms of cost-effectiveness, with other commonly endorsed health care interventions, when the prevalence of carotid stenosis is low. It may be most efficient, therefore, to refer for carotid imaging those with a high pretest probability of severe stenosis, e.g. patients with peripheral vascular disease or audible carotid bruits.

Figure

Figure 11. Cost effectiveness of 1) CUS with Angiographic Confirmation for Severe (70-99 percent) Stenosis (CUS/Angio-70 percent), and 2) MRA with Direct Referral to CEA for Severe Stenosis (MRA-70 percent), as a Function of (more...)