Summary of Findings

All-cause 30-day readmission is a moderate-strength indicator of 30-day mortality for coronary artery bypass grafting (CABG). Thirty-day readmission is a weaker indicator of early mortality for hip replacement. The correlations with early mortality are modest (eg, R range, 0.32 to 0.38; OR 1.14), but consistent. Thirty-day readmission would likely be feasible to collect as all community Medicare providers are already mandated to collect this information, but several potential limitations must be considered, such as lack of consensus on case-mix adjustment methods, whether to include socioeconomic status (SES), and difficulties capturing readmissions to non-index hospitals. Although the use of 30-day mortality as a measure of quality has recently been questioned, a recent VHA study reinforced its usefulness as a surrogate for long-term outcomes and disputed its susceptibility to gaming.

Adherence to standardized CABG wait time protocols, a set of cardiac surgery process measures, and a specific cardiac surgery antibiotic prophylaxis guideline-based protocol, but not other studied individual process measures, has decreased the likelihood of mortality and complications in cardiac surgery but not orthopedic surgery. We did not find any studies of performance measures collected in the inpatient setting that directly assessed the quality of postoperative bundles of care.

Implications for Policy and Implementation

Possible minimum requirements for Choice providers include acceptable performance on national rankings, compatible operational infrastructure, and ability to comply with an agreed-upon wait time threshold. In addition to these minimum requirements, the Office of Community Care could consider the added value of the performance measures that this review has identified as being indicators of desirable health outcomes. These include the single measures of 30-day mortality, a direct health outcome measure, and/or 30-day readmission, the indirect measure with the strongest association with mortality – both of which are commonly measured by surgery programs. Another option is to use a composite performance measure that includes mortality, readmission, and other process measures – such as the Society of Thoracic Surgeons' (STS) composite CABG measure. Also, we recommend use of public reporting program participation and measures of efficiency as additional considerations for Choice community provider minimum standards.

However, an unintended effect of stricter performance measure-based criteria for selective contracting with Choice providers may be an undersupply of providers, which could diminish Choice's effect on reducing Veterans' wait times.

PURPOSE

The ESP Coordinating Center (ESP CC) is responding to a request from the Office of Community Care for an evidence brief on the use of performance measures for selecting community cardiac and orthopedic surgical providers for the Veterans Choice Program. Findings from this evidence brief will be used to inform development of value-based community care purchasing pilots.

BACKGROUND

In 2014, after reports of long waiting times for VA clinical services,¹⁹ Congress passed the Veterans Access, Choice, and Accountability Act (VACCA), which expanded the criteria through which Veterans can access civilian providers.²⁰ VACCA mandated the VHA to implement a new national program within 90 days that allowed Veterans to pursue care from a third-party administrator’s (TPA) network provider in their community when VHA medical facilities have long wait times or when geographic accessibility is excessively burdensome (Veterans Choice Program or “Choice”). Important aspects of VACCA are that community providers must (1) “maintain the same or similar credentials and licenses as VA providers”, and (2) enter agreements with VA to furnish care rather than an open market or ‘voucher-based’ system.

Implementation of a new national program is a complex process, and Choice’s aggressive implementation timeline created challenges. Community provider network inadequacy was among a number of specific challenges identified by the VA Office of Inspector General review of the first 11 months of Choice implementation (November 1, 2014 through September 30, 2015).²¹ The VHA is planning to replace and expand existing community care networks to improve the quality of care for some services, such as general and vascular surgeries²² and Percutaneous Coronary Intervention (PCI).²³ Possible minimum requirements for Choice providers might include acceptable performance on national rankings, compatible operational infrastructure (eg, billing systems that can process bundled payments), and ability to comply with an agreed-upon wait time threshold. In addition to such requirements, the Office of Community Care is considering using performance measures to select and monitor community providers.²⁴ Initial pilots are targeting orthopedic and cardiac surgery because they are frequently performed and relatively standardized.

SCOPE/ANALYTIC FRAMEWORK

The underlying logic that guided our review is that the impact of using performance measures as criteria for selecting providers depends on whether the measures are indicative of lower complication and readmission rates, improved long-term survival and quality of life, and have acceptable measurement burden and unintended effects. The analytic framework (Figure 1) visually depicts this logic. The Key Questions and Inclusion Criteria define our specific focus within this framework.

Figure 1

Analytic Framework.

Key Question 1: When used as criteria for selecting providers, are certain performance measures valid indicators of surgical quality?

1. Is satisfactory performance on certain process standards associated with improved complications, lower readmissions, early survival, and long-term outcomes?
2. Are reduced readmission rates associated with improved early survival and long-term outcomes?
3. Is improved early survival associated with improved long-term outcomes?

Key Question 2: What is the comparative measurement burden of different performance measures?

Key Question 3: What are the unintended effects of using performance measures for selective contracting of community providers?

ELIGIBILITY CRITERIA

The ESP included studies that met the following criteria:

• Surgical services: Orthopedic and cardiac surgery services. We prioritized non-emergent total hip replacement, total knee replacement, and coronary artery bypass surgeries (CABG) as those are the most common VASQIP-assessed procedures.
• Health outcome performance measures: Early readmission rates, mortality rates
• Process performance measures: Post-operative plan of continuing care (eg, rehabilitation plan), wait times, adherence to surgical standards (eg, guideline compliance, infection-related outcome sets (eg, SCIP))
• Measurement burden: IT environment characteristics (eg, automated methods for collecting and auditing data, certification, interoperability), public reporting, audit and feedback, and electronic decision-support tools, chart review
• Unintended effects: Gaming, risk-based patient selection, appropriateness of care, changes in disparities, spillover effects, et cetera
• Timing: Any study follow-up durations
• Setting: Any
• Study design: Any, but may prioritize to accommodate timeline using a best-evidence approach

LITERATURE FLOW

The literature flow diagram (Figure 2) summarizes the results of the search and study selection processes.

Figure 2

Literature Flowchart.

Searches resulted in 3,519 unique potentially relevant articles. Overall, we included 38 studies: 6 systematic reviews^60-65 one randomized controlled trial,⁶⁶ and 31 observational studies.^{1-12,14-18,67-80} Only 4 of the observational studies were prospective cohorts,^8,14,17,66 with the rest being retrospective. Eleven studies addressed adherence to a surgical standard,^{10-12,14-16,68,71,72,74,76} 7 addressed readmission,^3-9 18 addressed wait times,^{17,18,60-67,69,70,73,75,77,79-81} and 2 addressed mortality.^1,2 Detailed reasons for study exclusion are provided in Appendix C in the supplemental materials. More than half of the studies were multicenter (66%) and within the US (53%), but only 44% used data collected in the last decade. Median sample size was 2,218 (range 47 to 2,121,215). Of the studies we formally quality assessed, most were good^2-4,7-12,72 or fair quality.^1,5,6,14-18 We rated 4 studies as poor quality for lack of adjustment for potential confounders, lack of information on missing data, and/or potential risk of selection bias.^68,71,74,76 Sixteen studies had high levels of adjustment for patient-, procedure-, and provider-level factors,^{2-5,7-12,14,16,18,66,72,81} while 7 studies adjusted for patient-level factors only,^{1,6,15,17,67,74,76} and 2 studies did not adjust for any confounding factors.^68,71 We did not formally assess the quality of the systematic reviews or 7 studies in orthopedic fracture populations,^{69,70,73,75,77,79,80} as these emergency populations were of less direct relevance to this review (full details of data abstraction and quality assessment in Appendix D in the supplemental materials).

KEY QUESTION 1A. Is satisfactory performance on certain process standards associated with improved complications, lower readmissions, early survival, and long-term outcomes?

Wait Time

Adherence to standardized CABG wait time protocols shows a decrease in the likelihood of mortality.^17,18 We found no studies on wait times for total knee replacements or total hip replacements. For emergent hip fracture repair in the elderly, delays beyond 24-48 hours are associated with increased mortality and complications, but this evidence likely has limited applicability to the non-emergent, elective surgery populations of greatest interest in this review.

Cardiac Surgery

Delays exceeding British Columbia provincial wait list guidelines (6 weeks for semiurgent and 12 weeks for nonurgent) were associated with increased probability of in-hospital death based on findings from a large, nationally representative retrospective cohort studiy in a cardiac surgery population awaiting CABG (Table 1).¹⁸ This study evaluated the association of adherence to 2 separate established wait time guidelines – British Columbia provincial and the stricter Canadian Cardiac Society (CCS) (2 weeks for semiurgent and 6 weeks for nonurgent) – and in-hospital mortality by comparing outcomes across 3 categories of delays: short (within CCS timeframes), prolonged (longer than CCS but within provincial targets), and excessive (longer than both guidelines). Adherence to at least the provincial guidelines could potentially decrease risk of in-hospital mortality, as the short CCS timeframes had a protective effect compared with nonadherence with both guidelines, but not compared to the provincial guideline adherence. The validity of adherence to a strict wait time protocol as an indicator of quality of care is also supported by the finding that use of a wait-time triage system to assign standard wait times of 0 days for emergent cases, 7 days for in-hospital urgent, 21 days for out-of-hospital semi-urgent A, and 56 days for semi-urgent B resulted in equitable mortality and morbidity across waiting groups.¹⁷ Although the supporting study likely had limited statistical power and applicability because it focused on a small sample of patients with left main coronary artery disease from a single hospital in Halifax, its approach to evaluating the safety of using a standardized triage system for determining CABG wait times is an important mechanism for evaluating the validity of wait time protocols as quality indicators.¹⁷ No studies reported on readmission, complication or other outcomes of interest. Both studies^17,18 adjusted for patient, clinical, and surgical factors (see appendix D), but were conducted outside the US, so their applicability to the US health care system is limited.

Table 1

Cardiovascular Surgery Wait Times and Patient Outcomes.

Emergent Fracture Repair

Early surgery (usually within 24-48 hours) was associated with reduced mortality (OR 0.74, 95% CI 0.67 to 0.81) in elderly patients with hip fractures needing emergent repair according on the best evidence from a 2012 good-quality systematic review of 35 studies.⁶² However, it is important to note that adjusted and unadjusted data were combined, and the cut-off time for wait time and mortality varied by study. When stratified by cut-off time for surgical delay (<12, <24, <48, <96 hours), results showed a significant association between wait time and mortality for wait times within 24-48 hours, but no association betweeen other cut-off times. Among systematic reviews that looked at complications in this population,^61,64,65 the most recent good-quality systematic review⁶⁵ found early surgery (within 24-48 hours) to be associated with decreased rates of pneumonia (RR 0.59; 95% CI 0.37 to 0.93; 2 studies; N=2793) and pressure sores (RR 0.48; 95% CI 0.34 to 0.69; 3 studies; N=3023), but not deep vein thrombosis (RR 0.97; 95% CI 0.56 to 1.68; 2 studies; N=4679) or pulmonary embolism (RR 0.66; 95% CI 0.17 to 2.58; 2 studies; N=2822). The study authors caution that their confidence in the finding is limited because the included primary studies did not adjust for potential confounding factors.⁶⁵

Findings from subsequently published primary literature neither confirm nor refute these findings due to their serious methodological limitations or use of a prolonged follow-up period.^67,75,81 Wait times exceeding 24 hours were not associated with 1-year mortality in a sample of 567 people admitted to a single orthogeriatric unit in Oslo.⁶⁷ This study was underpowered to find a difference; post-hoc power for the study was 42.2% and revealed that a future study to would require 1,249 patients to reach adequate power.⁸² In a sample of 828 patients in Italy, wait times exceeding 48 hours were not associated with 2-year mortality.⁷⁵ A strength of this study was its adjustment for comorbidities. But it is unclear how comparable findings from a 2-year follow-up period are to the most recent systematic review⁶² in which the mean follow-up period was unspecified, but most likely shorter. Finally, the most recent study⁸¹ found no association between wait time and mortality in the elderly hip fracture population. However, we have limted confidence in its findings due to the lack of adjustment for any potential confounders and small sample size.

In hip fracture repair populations not restricted to the elderly, one good-quality systematic review found that the majority of studies (14/24) that adequately controlled for confounding did not find a correlation between wait time and mortality, but found that delays increased risk of complications.⁶⁰ Results were consistent regardless of variation in adjustment, cut-off time for wait time, and cut-off time for mortality. Among 4 subsequently published primary studies,^66,70,73,77 results from the single study⁷⁷ that adequately controlled for potential confounding (adjusted for age, gender, race, comorbidity burden, insurance status, day of admission, hospital size, teaching status, and region) were consistent with and strengthen the findings of the systematic review (Table 2).

Table 2

Wait Time and Patient Outcomes in Orthopedic Surgery (Studies not captured in systematic reviews).

In patients undergoing emergent ankle fracture repair, delayed surgery (>24 hours, up to 14 days) significantly increased risk of wound complications in 6 of 11 studies.^63,79 Although we did not formally rate the quality of these studies in this lower-priority population, the strength of evidence is likely very low because they were small – sample sizes < 100 – and did not control for potential confounding.^63,79

In patients with multiple fractures, one retrospective cohort of 1005 patients at a single Level 1 trauma center provides low-strength evidence that performing fixation of unstable axial fractures within 24 hours of the trauma significantly reduced risk of complications.⁸⁰ The applicability of this evidence to the non-emergent surgeries of interest is likely low.

Surgical Standards

Low ‘total care quality’ scores moderately increased risk of 30-day mortality and complications following CABG (OR range, 1.51 to 1.91; low strength of evidence; 2 studies).^10,12 The ‘total care quality’ score ranges from 0 to 5-6, where 0 means no process measures were met and 5-6 is the total number of process measures used. Adherence to individual cardiac surgery process measures was generally not associated with reduced mortality and complications, except that violation of an antibiotic prophylaxis guideline-based protocol increased risk of mortality and surgical site infections (OR range 7.03 to 10.16; low strength of evidence; one study).¹⁴ For orthopedic surgery, among 4 studies the best studies provided low-strength evidence that a composite quality score of CMS NQF measures, and adherence to individual SCIP measures were not valid indicators of either mortality or complications.^15,16

Cardiac Surgery

In cardiac surgery populations, composite scores of ‘total care quality’ (ie, total number of individual process measures missed) may better predict patient health outcomes than adherence to an individual process measure (Table 3). Two good-quality studies examined adherence to surgical quality measures in cardiac surgery using a composite quality score based on the total number of recommended NQF or SCIP measures achieved.^10,12 One study of 81,289 CABG patients found increased 30-day mortality with increasing proportion of patients who failed to receive recommended Surgical Care Improvement Project (SCIP) measures.¹⁰ This study found no association between the number of missed measures and readmission.¹⁰ Another study of 2,218 CABG patients found higher odds of several complications for patients with a low quality score based on the number of National Quality Forum (NQF) process measures achieved (range 0 to 5, 5 = high quality care, all other scores = low quality care).¹²

Table 3

Adherence to Surgical Standards and Patient Outcomes in Cardiac Surgery.

Among individual cardiac surgery process measures studied (Table 3),^11,13,14,74 only adherence to an antibiotic prophylaxis guideline-based protocol improved outcomes.¹⁴ Initiating the first skin incision for elective cardiac surgery before the end of vancomycin infusion was associated with large increased risk of surgical site infection rates and mortality in-hospital or within 30 days in a prospective study of 741 adults from a single center in a 1200-bed tertiary care university hospital in Italy.¹⁴ Three other studies investigated adherence or lack of adherence to a single measure, pre-operative beta-blocker use¹¹ and blood glucose maintenance.^13,74 These studies found no association between adherence to surgical standards and 30-day mortality or complications.^11,13,74 An additional pre-post study found decreased infection rates after implementation of an enhanced targeted infection control program, including education, improved hygeiene, MRSA screening, care pathways, and antimicrobial surgical prophlaxis. However, this study was limited by no control for potential confounders and did not specifically assess adherence to the antimicrobial surgical prophylaxis.⁶⁸ These results indicate that composite measures or ‘all or none’ measures of surgical standard adherence may better measure overall quality of care, as they evaluate the overall system of care for several processes. In contrast, adherence to single surgical standards may be influenced by the practices of single team members.⁸³

These studies were generally of good quality, using rigorous methods to adjust for potential confounding, and 3 out of the 7 studies were multicenter.^10-12 The exceptions were 2 studies that inadequately addressed potential confounding factors, either only controlling for a single factor or not controlling for any potential confounders.^68,74 As most of these studies were in CABG patients – which is a well-developed and frequently performed surgery – these results might not be applicable to other types of cardiac surgery.¹²

Orthopedic Surgery

Adherence to orthopedic surgical standards was consistently not associated with health outcomes in all of 4 included studies.^15,16,71,76 One study examined hospital quality in hip or knee replacement surgery in 4 performance tiers based on a composite quality score of CMS NQF measures.¹⁵ This composite score included process measures (adherence to antibiotic prophylaxis standards, etc), as well as outcome measures (readmission avoidance, etc), but the differences in the top 20% of hospitals based on performance and the remaining hospitals were driven by the process measures. There was no significant difference in mortality across hospital tiers, but a trend toward a higher rate of mortality in tier 4 hospitals, classified as the bottom 50% of performance (r = 0.116, P = 0.088). Additionally, complication rates did not differ significantly by hospital tier, and readmission avoidance did not differ significantly between the top 20% of hospitals and all other hospitals.¹⁵ Another study examined level of compliance to SCIP measures (eg, ‘highly compliant’ indicated greater than median level of compliance) and complications among hip arthroplasties at 128 New York state hospitals.¹⁶ This study found no association of adherence to the individual SCIP measures and post-operative infection rates or hospital-level SSI rates, with the exception of increased infection rates with higher adherence to the SCIP “VTE-2” prevention measure, which measures the percentage of hospital patients who received appropriate venous thromboembolism prophylaxis within 24 hours prior to surgery to 24 hours after surgery (post-operative infection: OR 1.50, 95% CI 1.07 to 2.12; hospital-level SSI: OR 1.91, 95% CI 1.31 to 2.79).

Two studies examined rates of complications (SSI) before and after implementation of surgical standards.^71,76 One poor-quality study with no adjustment for potential confounders examined adherence to surgical antibiotic prophylaxis (SAP) among 6 hospitals in Korea before and after implementation of a national hospital evaluation program.⁷¹ Although adherence to SAP improved (P < 0.01), there were no significant changes in SSI rate for hip or knee arthroplasty or spine surgery. Another poor-quality study with minimal adjustment for confounders examined SSI rates before and after implementation of SCIP in a single institute in the US.⁷⁶ After implementation, the SCIP rate adherence was more than 98%. Multivariate analysis (adjusting for surgery and SCIP factors) showed that SCIP adherence was associated with increased risk of superficial infection (estimate: -4.45, P < 0.001), and pulmonary embolism (estimate: 0.34, P < 0.001), but there were no significant associations with deep infection (P = 0.46) or deep vein thrombosis (P = 0.51). These findings suggest that adherence to surgical standards may not predict surgical complication rates. However, these studies are limited by lack of a concurrent control, minimal or no adjustment for potential confounding variables, and limited information on processes prior to implementation of the surgical standard programs.

KEY QUESTION 1B. Are reduced readmission rates associated with improved early survival and long-term outcomes?

For CABG surgery, the 3 most rigorous studies^3,4,7 found a modest but consistent significant association between 30-day readmissions and 30-day mortality (r range, 0.32 to 0.38; OR 1.14; moderate strength of evidence), but not with process measures. For hip replacement, there was a correlation between 30-day readmission and mortality (one study, low strength of evidence),⁷ but the association between readmission rates and process measures was inconsistent. Among 2 studies,^7,9 readmission rates were only correlated with aggregate process compliance measures in the study with a broader orthopedic population that had wider variation in readmission rates.⁹

Coronary Artery Bypass Grafting (CABG)

30-day Readmission Association with 30-day Mortality

Among 4 retrospective cohort studies, the 3 most rigorous studies^3,4,7 found a modest but consistent significant association between 30-day readmissions and 30-day mortality (r range, 0.32 to 0.38; OR 1.14) (Table 4).^3-5,7 The most broadly applicable data comes from the only study using a national data source.⁷ Others used data from the states of New York,^3,4 or California.⁵ All used risk-adjusted readmission and mortality variables, but each used slightly different analytic approaches. For example, the national Medicare data study used a standardized readmission rate whereby they calculated the composite readmission rate for each hospital by averaging the observed-to-expected (O/E) readmission ratios for the 6 procedures for the hospital and weighting each ratio on the basis of the number of cases for that procedure. Then they compared the standardized risk-adjusted readmission rates between hospitals in the highest quartile for mortality to the lowest quartile.⁷ In contrast, the California study observed trends in discordance among hospitals that were O/E outliers (95% CI excluded 1.0) for mortality or readmission and not both (“discordant”).⁵ Regardless, 3 of the 4 studies found a significant association between 30-day readmission and 30-day mortality.^3,4,7 We have lower confidence in the validity of the outlier study, however, due to their use of weaker statistical methods.⁵ Authors of the outlier study noted that most hospitals had concordant O/E readmission and mortality rates, which “points to the likely existence of some association between readmission and mortality rates.” But, they based their conclusions on the trends they observed among the outliers. Among outliers, 85% were “discordant” for readmission and mortality (for example, hospitals with high readmission and low mortality). This approach may exaggerate the significance of the outliers as it neglects the statistical effects among the data as a whole, which was predominantly comprised of concordant hospitals.

Table 4

30-day Readmission in Coronary Artery Bypass Grafting (CABG) Patients.

Readmission Associated with 100-day Mortality

One retrospective cohort study evaluated the association between longer-term readmission rates (within 100 days) and 30-day mortality among 14 hospitals in Israel in 1994.⁸ Although this study found that high mortality-ranked hospitals had higher rates of readmission (OR 1.34, P = 0.003), this finding likely has low relevance to contemporary care in the US.

Readmission Associated with Process Measures

Among 3 retrospective cohort studies that used diverse process measure assessment,^6,7,9 none found a significant association with readmission. Process measure compliance assessment ranged widely and included (1) peer-reviewing charts based on a set of Health Care Financing Administration (HCFA)-specified generic quality screens to evaluate care provided as acceptable or problematic,⁶ (2) evaluating compliance with surgery-specific SCIP process measures (overall compliance with all 9 process measures, or number of compliant events divided by the number of opportunities to provide recommended care (ie, non-smoker not eligible for smoking sessation counseling),⁹ and (3) comparing hospitals in the highest and lowest decile of the Hospital Quality Alliance (HQA) surgical score (composite of 9 process measures).⁷ Some authors speculated that the lack of association may be because these process measures represent just a fraction of the entire spectrum of care, and do not capture other important features such as care coordination at the time of discharge.⁹

Readmission associated with composite process/outcome measures

Lower readmission rates were weakly correlated (Spearman rank correlation= –0.154) with higher composite scores (including mortality, major morbidity, internal mammary artery graft, and NQF-endorsed perioperative medications) in a secondary subgroup analysis of 827 CMS CABG providers from the 2010 STS database.⁸⁸ However, we have insufficient information to determine the strength of this evidence as this finding was only very briefly noted in the discussion section of the main study, which was devoted to the development of the readmission measure. No other information about the methodology were provided in the publication or via author request.

KEY QUESTION 1C. Is improved early survival associated with improved long-term outcomes?

KEY QUESTION 2. MEASUREMENT BURDEN

Performance measures vary in their burden of data collection and reporting. Measurement burden is an important consideration, as successful implementation depends on realistic attainability.⁸⁹ However, we found no studies that formally evaluated comparative measurement burden. In general, measures such as mortality and readmission are often more appealing because they are the most feasible to collect as they are already being regularly collected and reported for internal and public quality improvement initiatives. However, they have the potential disadvantages of insufficiently discriminating quality-sensitive, preventable events. However, determining preventability would require committee or chart audits and would likely be resource-prohibitive.¹ Also, risk-adjustment is necessary, but administrative databases may not capture all relevant variables⁶ and there is little consensus on which methods to use.³² Surgical standard adherence, postoperative plan of care, and wait times are likely associated with greater measurement burden. This is because (1) there is more variability in whether any are measured and, if so, which ones; (2) if tracked electronically, sources may have low reliability;¹⁰ and (3) they may require additional staff to set up and implement,⁷⁴ possibly through chart review.⁸⁵ Additionally, wait times may present additional barriers to collection. Different wait time standards may be needed for different subgroups, as safe wait times may vary by patient subgroups and surgery types.⁶⁹ There is a risk that clinically-justified longer wait times may be penalized if medical records and/or databases lack a mechanism for recording supporting documentation. For example, frailer patients may be more likely to be delayed due to need for stabilization prior to surgery.^60,62

KEY QUESTION 3. UNINTENDED EFFECTS

Concern has been expressed that focus on specific performance measures may result in unintended effects such as gaming, risk-based patient selection, appropriateness of care, changes in disparities, and spillover effects. For example, 30-day readmission may be gamed by delaying readmission to 31 days instead of 30 days.⁹⁰ Or for 30-day mortality, intensive care management and end-of-life care may be modified against the patient's best interest to extend death beyond 30 days.² Also, adherence to some surgical standards is debated for some patients (ie, beta-blockers for specific cases of off-pump revascularization) and adherence may harm those patients.¹¹ For wait times, as demonstrated in the Phoenix VA Medical Center in 2014, use of a 2-week wait time performance measure – regardless of clinical need, facility capacity, and other accommodations such as secure messaging or telehealth – resulted in the manipulation of scheduling practices in order to meet performance goals.⁴⁸ Finally, an unintended effect of using any of these performance measures for selective contracting with Choice providers is that it may result in an undersupply of providers, which could diminish Choice's intended effect on reducing Veterans' wait times.

Evaluation of such unintended consequences has largely been lacking. Recent systematic reviews of using various performance measures in value-based payment programs found very limited evidence assessing the extent of gaming, inconsistent evidence of effects on health disparities, and only some evidence of both positive and negative effects on unincentivized measures.⁹¹ The only additional evidence we found was from a recent VHA study that examined surgical mortality between postoperative days 25 and 35 and found no evidence of delayed mortality within the highest readmission decile.²

Limitations of 30-day Readmission Rates

Although evidence suggests 30-day readmission is a moderately valid indicator of surgical quality and is likely reasonably feasible to assess, its limitations and potential consequences must be considered. A systematic overview of methodological aspects of readmission rates³² and several editorials have provided detailed evaluations of the limitations of 30-day readmissions.^{28,31,43,44,92,93} The most common criticism about all-cause readmission is that it is insensitive to true quality of care issues because only a minority are avoidable (∼27%),⁹⁴ with some being planned, and the majority being outside of the hospitals' control and likely driven by factors such as mental illness, poor social support, and poor community outpatient resources.²⁸ A proposed solution is to limit analyses to preventable/avoidable readmission.³² However, as doing so would require additional resources in the form of independent expert judgment, this is likely technically infeasible to implement. Also, our review identified evidence supporting all-cause readmission as a valid indicator of quality in the form of 30-day mortality. A second common criticism of readmission rates is that their validity is limited by variability in and lack of consensus on the methods used to adjust for case-mix. Although risk prediction models consistently adjust for some variety of patient-, surgical-, and hospital-level variables, current evidence provides limited guidance on which specific variables should be included because studies have been highly heterogenous in their methodology, patient groups, and considered variables, and have found different factors increased risk of readmission. One of the key controversies in the ongoing dialogue about risk adjustment methods is over whether to adjust for socioeconomic status (SES). CMS and NQF have argued not to include adjustment for SES in readmission risk adjustment, because factors such as SES, race, and gender may be associated with inequalities in care and these inequalities may be obscured if these factors are included in risk models. Others argue that differences in outcomes by SES may in fact be due to poorer living conditions, lack of support, and/or poorer nutrition, for which hospitals should not be accountable.⁹⁵Although there is much debate around this issue, it is unclear if adding SES to risk adjustment models would change actual hospital rankings.⁹⁶ Third, although not necessarily unique to readmissions, the lack of widely acceptable thresholds for minimum performance may be a barrier to widespread use.⁴⁴ Although thresholds must be set, there is an inherent risk of misclassifying providers or otherwise unfairly excluding providers with lower performance for idiosyncratic reasons. Finally, concern has been expressed that focus on readmission may result in neglect of other important aspects of care quality and/or gaming to achieve lower rates at the expense of patient care (ie, admitted at 31 days instead of 30 days). Although evaluation of such unintended consequences is largely lacking, a recent VHA study that examined surgical mortality between postoperative days 25 and 35 by risk decile showed no evidence of delayed mortality within the highest readmission decile.² Also, as participation in the Choice network would be voluntary and not universally imposed on the community at large, this may minimize potential unintended consequences of readmission. However, adding performance measures to Choice network participation eligibility requirements may be seen as an additional barrier for community providers and may perpetuate the current network inadequacy challenges.²¹

General Limitations

The main general limitations of the evidence included understudied measures, methodological features, and applicability. With few exceptions,^1,2 evidence on the validity of the surgical performance measures evaluated in this review has largely neglected evaluating their association with the most patient-centered outcomes of long-term health outcomes. Also, although care coordination and follow-up care interventions have been shown to impact readmissions and there has been a call for development of innovative care transitions and postoperative care plan measures, other than one study of time to follow-up appointment timing (before or after the recommended 3 weeks),⁸⁵ evidence is lacking on potential postoperative plan of care measures such as presence of pre-discharge assessment and ensuring every patient is scheduled for a follow-up visit. Also, as mentioned above, although there is much concern about the unintended consequences of using performance measures for various purposes, little research has been done in this area. Methodologically, the main strengths of these studies are that the reliability of the data sources was generally high, in that they were from established and regularly audited quality improvement databases, adequacy of adjustment for potential confounding was at least moderate, in that most at least adjusted for patient-level factors, and they likely had adequate statistical power with a median sample size of 2,218 (range 47 to 2,121,215). The main methodological weaknesses were that all but 4 were retrospective, none was able to capture events that may have occurred at hospitals other than the index hospital, and most were based on administrative databases that did not include information not captured by billing codes, such as disease severity. Although a majority of studies were within the US and multicenter, applicability is still limited because most focused on CABG and over half involved samples more than a decade old.

The main limitations of our review methods include (1) our literature search, (2) our focused scope, and (3) our use of sequential instead of independent dual assessment. For our literature search, although we searched 5 databases and other sources and used a comprehensive search strategy, the inconsistent terminology used in literature on performance measures and quality of care may have increased our risk of missing relevant studies. Second, to meet our condensed timeframe we focused our scope to the highest-priority populations and measures of the Office of Community Care. However, this limits the applicability of our findings to other populations and measures of interest. Third, although sequential dual review is a widely used method, its comparison to independent dual review has not yet been empirically studied and may have increased the risk of error and bias.

Future Research

To improve evidence about the validity of performance measures as indicators of surgical quality and increase its applicability to broader populations, high priorities for future research include (1) prospective evaluation of a broader array of modern surgical populations; (2) improving consensus on case-mix adjustment methods, which could be in the forms of an updated systematic review, as well as a new primary study that compares the use of various adjustment methods; (3) evaluation of associations with longer-term patient-important outcomes; (4) evaluation of unintended consequences including gaming and risk-based patient selection, evaluation of appropriateness of care, changes in disparities, and spillover effects;⁹⁷ and (5) evaluation of postoperative plan of care measures such as presence of pre-discharge assessment and ensuring every patient is scheduled for a follow-up visit.

As the reason for this review was to identify performance measures that are valid indicators of quality, for implementation in selective contracting with Choice community providers to improve quality of care for Veterans, implementation should be accompanied by plans not only to extend the evidence on validity, but also to evaluate the effectiveness of the implementation, including how selective contracting is affecting community providers' experience. For the purposes of this review, although the primary interest in the validity of performance measures is in their use for selective contracting, likely they will continue to be used to monitor Choice patient outcomes as well for general quality improvement. Currently, data is routinely collected from community providers using a one-page form that is scanned into the Veterans' charts. However, this process does not permit systematic electronic querying for data analysis. To improve capabilities to monitor Veterans' outcomes in the community and compare to care within VHA, VA Secretary Shulkin and others have called for creation of data standards and standardized electronic data systems that would better permit aggregation of data across sites.^19,44

Implications for Policy and Implementation

Possible minimum requirements for Choice providers include acceptable performance on national rankings, compatible operational infrastructure, and ability to comply with an agreed-upon wait time threshold. In addition to these minimum requirements, the Office of Community Care could consider the added value of the performance measures that this review has identified as being indicators of desirable health outcomes. These include the single measures of 30-day mortality, a direct health outcome measure, and/or 30-day readmission, the indirect measure with the strongest association with mortality – both of which are commonly measured by surgery programs. Another option is to consider use of a composite performance measure that includes mortality, readmission, and other process measures – such as STS's composite CABG measure. We also recommend use of public reporting program participation and measures of efficiency as additional considerations for Choice community provider minimum standards. However, an unintended effect of stricter performance measure-based criteria for selective contracting with Choice providers may be an undersupply of providers, which could diminish Choice's effect on reducing Veterans' wait times.

An alternative to using individual performance measures that are direct or indirect indicators of health outcomes is to use a composite performance measure³⁶ that includes mortality, readmission, and other process measures – such as STS's composite CABG measure. Composite performance measures are meant to broadly encapsulate the overall quality of care by combining information from multiple individual outcome and process performance measures into a single, comprehensive, multidimensional measure. An advantage of composite measures is that they can reduce data burden by translating information on a broad range of indicators that may not otherwise be possible to track. However, as with any composite performance measure, including intelligence quotients, there are a few challenges to consider. Use of a single composite measure may lose important detail on variation in performance across individual component measures. For example, an intermediate score on a composite measure may reflect excellent performance on some measures and below-average on others. Also, all users' unique interests, values, and preferences may not be reflected by the subjective relative weighting of different components in scoring composite measures. Ideally, choice of a composite performance measure should consider the strength of the association between its components and health outcomes, the transparency and comprehensibility of its inputs and rules, the regularity of updating to maintain clinical relevance as new information becomes available, and its reliability and validity. The Office of Community Care could consider use of a composite measure as a highly feasible and comprehensive approach to determine eligibility of community providers if they determined that the potential advantages of a composite measure outweighed its challenges and identified a rigorously developed and validated composite measure that is widely accepted and used.

We examined whether performance measures meant as indirect indicators of health outcomes are related to health outcomes and whether they are difficult (feasible) to measure, but not whether potential Choice community providers collect and report valid data. To apply performance measurement-based selective contracting, VA must be able to determine whether the providers' data are reliably measured. Accordingly, in addition to selecting providers based on meeting thresholds for acceptable 30-day readmission and 30-day mortality rates, decision-makers may also require Choice community providers to participate in a public reporting program that involves periodic auditing, including publically available claims databases such as CMS, commercial comprehensive clinical databases such as Vizient, Crimson, Premier, etcetera, or specialty registries such as STS, ACC, or NSQUIP. This would ensure the reliability of Choice community providers' performance measures, and the participation in public reporting itself may also be a strong motivator for quality improvement. Preferable characteristics of performance reporting programs include (1) use of defined populations, (2) longer time in operation, (3) transparency through providing data that identifies specific providers/surgical centers, and (4) evaluation across the continuum of care.

Also, although we did not evaluate measures such as length of stay and patient visits per physician per month, we suggest decision-makers consider the usefulness of such efficiency measures as a supplement to indicators of patient outcomes. For providers that meet minimum standards on direct or indirect health outcome measures, choosing more efficient providers could benefit national efforts to improve health care value.

CONCLUSIONS

Among performance measures meant as indirect indicators of health outcomes, 30-day readmission is the strongest indicator of 30-day mortality for CABG and for hip replacement and is feasible to measure. Its use for selecting Choice providers is reasonable, but its potential limitations must be considered. Contrary to recent criticism, the 30-day mortality measure is likely a valid surrogate for long-term survival, with a lower than expected risk of gaming. Use of a robust and widely used composite measure of direct and indirect indicators of health outcomes may also be a highly feasible and comprehensive approach to determining eligibility of Choice providers. Also, we recommend use of public reporting program participation and measures of efficiency as additional considerations for Choice community provider minimum standards. An unintended effect of using performance measures as criteria for selecting providers in general is that it may result in an undersupply of Choice providers that could diminish Choice's intended effect of reducing Veterans' wait times.

APPENDIX A. EXISTING PERFORMANCE MEASURE SETS

APPENDIX B. SEARCH STRATEGIES

APPENDIX C. LIST OF EXCLUDED STUDIES

Exclude reasons: B=Relevant for background information only, 1=Ineligible population, 2=Ineligible intervention, 3=Ineligible comparator, 4=Ineligible outcome, 5=Ineligible timing, 6=Ineligible study design, 7=Ineligible publication type 8=Outdated or ineligible systematic review, 9=Non-English language

APPENDIX D. EVIDENCE TABLES

APPENDIX E. PEER REVIEW

REFERENCES

1.

Centers for Medicare & Medicaid Services. Consensus core set: Orthopedic measures. 2015; https://www​.cms.gov/Medicare​/Quality-Initiatives-Patient-Assessment-Instruments​/QualityMeasures​/Downloads​/Orthopedic-Measures.pdf. Accessed May 9, 2017.

2.

Centers for Medicare & Medicaid Services. Consensus core set: Cardiovascular measures. 2016; https://www​.cms.gov/Medicare​/Quality-Initiatives-Patient-Assessment-Instruments​/QualityMeasures​/Downloads​/Cardiovascular-Measures.pdf. Accessed May 9, 2017.

3.

National Quality Forum. National quality forum: Measure search. 2017; http://www​.qualityforum​.org/Qps/QpsTool.aspx?m​=2515&e=1. Accessed May 9, 2017.

4.

Khuri SF, Daley J, Henderson W, et al. The department of veterans affairs' nsqip: The first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National va surgical quality improvement program. Annals of Surgery.1998;228(4):491. [PMC free article: PMC1191523] [PubMed: 9790339]

5.

Khuri SF, Henderson WG, Daley J, et al. The patient safety in surgery study: Background, study design, and patient populations. Journal of the American College of Surgeons. Jun 2007;204(6):1089-1102. [PubMed: 17544068]

6.

Dixon JL, Papaconstantinou HT, Hodges B, et al. Redundancy and variability in quality and outcome reporting for cardiac and thoracic surgery. Baylor University Medical Center Proceedings. 2015;28(1):14-17. [PMC free article: PMC4264699] [PubMed: 25552787]

7.

Shiloach M, Frencher SK, Steeger JE, et al. Toward robust information: Data quality and inter-rater reliability in the american college of surgeons national surgical quality improvement program. Journal of the American College of Surgeons. 2010;210(1):6-16. [PubMed: 20123325]

8.

Fink AS, Campbell DA, Jr., Mentzer RM, Jr., et al. The national surgical quality improvement program in non-veterans administration hospitals: Initial demonstration of feasibility. Annals of Surgery. Sep 2002;236(3):344-353; discussion 353-344. [PMC free article: PMC1422588] [PubMed: 12192321]

9.

Fry DE. Surgical site infections and the surgical care improvement project (scip): Evolution of national quality measures. Surgical Infections. Dec 2008;9(6):579-584. [PubMed: 19216670]

10.

Leung F, Lau TW, Kwan K, Chow SP, Kung AW. Does timing of surgery matter in fragility hip fractures? Osteoporos Int. Dec 2010;21(Suppl 4):S529-534. [PMC free article: PMC2974918] [PubMed: 21057992]

11.

Moja L, Piatti A, Pecoraro V, et al. Timing matters in hip fracture surgery: Patients operated within 48 hours have better outcomes. A meta-analysis and meta-regression of over 190,000 patients. PLoS One. 2012;7(10):e46175. [PMC free article: PMC3463569] [PubMed: 23056256]

12.

Shiga T, Wajima Z, Ohe Y. Is operative delay associated with increased mortality of hip fracture patients? Systematic review, meta-analysis, and meta-regression. Canadian Journal of Anaesthesia. 2008;55(3):146-154. [PubMed: 18310624]

13.

Simunovic N, Devereaux PJ, Sprague S, et al. Effect of early surgery after hip fracture on mortality and complications: Systematic review and meta-analysis. CMAJ. Oct 19 2010;182(15):1609-1616. [PMC free article: PMC2952007] [PubMed: 20837683]

14.

Khan SK, Kalra S, Khanna A, Thiruvengada MM, Parker MJ. Timing of surgery for hip fractures: A systematic review of 52 published studies involving 291,413 patients. Injury. Jul 2009;40(7):692-697. [PubMed: 19450802]

15.

Schepers T, De Vries MR, Van Lieshout EM, Van der Elst M. The timing of ankle fracture surgery and the effect on infectious complications; a case series and systematic review of the literature. International Orthopaedics. 2013;37(3):489-494. [PMC free article: PMC3580081] [PubMed: 23288046]

16.

Légaré JF, MacLean A, Buth KJ, Sullivan JA. Assessing the risk of waiting for coronary artery bypass graft surgery among patients with stenosis of the left main coronary artery. CMAJ. Aug 16 2005;173(4):371-375. [PMC free article: PMC1188222] [PubMed: 16103509]

17.

Sobolev BG, Fradet G, Kuramoto L, Rogula B. An observational study to evaluate 2 target times for elective coronary bypass surgery. Medical Care. Jul 2012;50(7):611-619. [PubMed: 22525613]

18.

Meessen JM, Pisani S, Gambino ML, et al. Assessment of mortality risk in elderly patients after proximal femoral fracture. Orthopedics. Feb 2014;37(2):e194-200. [PubMed: 24679208]

19.

Holvik K, Ranhoff AH, Martinsen MI, Solheim LF. Predictors of mortality in older hip fracture inpatients admitted to an orthogeriatric unit in oslo, norway. Journal of Aging & Health. Dec 2010;22(8):1114-1131. [PubMed: 20881106]

20.

Karademir G, Bilgin Y, Ersen A, et al. Hip fractures in patients older than 75 years old: Retrospective analysis for prognostic factors. International Journal Of Surgery. Dec 2015;24(Pt A):101-104. [PubMed: 26563486]

21.

Clague JE, Craddock E, Andrew G, Horan MA, Pendleton N. Predictors of outcome following hip fracture. Admission time predicts length of stay and in-hospital mortality. Injury. Jan 2002;33(1):1-6. [PubMed: 11879824]

22.

Griffiths EJ, Cash DJ, Kalra S, Hopgood PJ. Time to surgery and 30-day morbidity and mortality of periprosthetic hip fractures. Injury. Dec 2013;44(12):1949-1952. [PubMed: 23639824]

23.

Lund CA, Moller AM, Wetterslev J, Lundstrom LH. Organizational factors and long-term mortality after hip fracture surgery. A cohort study of 6143 consecutive patients undergoing hip fracture surgery. PLoS ONE [Electronic Resource]. 2014;9(6):e99308. [PMC free article: PMC4057264] [PubMed: 24926876]

24.

Lurati Buse GA, Bhandari M, Devereaux P, Inverstigators T. Accelerated care versus standard care among patients with hip fracture: The hip attack pilot trial. CMAJ. Jan 07 2014;186(1):E52-60. [PMC free article: PMC3883849] [PubMed: 24246589]

25.

Ryan DJ, Yoshihara H, Yoneoka D, Egol KA, Zuckerman JD. Delay in hip fracture surgery: An analysis of patient-specific and hospital-specific risk factors. Journal of Orthopaedic Trauma. Aug 2015;29(8):343-348. [PubMed: 25714442]

26.

Tennent TD, Calder PR, Salisbury RD, Allen PW, Eastwood DM. The operative management of displaced intra-articular fractures of the calcaneum: A two-centre study using a defined protocol. Injury. Jul 2001;32(6):491-496. [PubMed: 11476816]

27.

Vallier HA, Super DM, Moore TA, Wilber JH. Do patients with multiple system injury benefit from early fixation of unstable axial fractures? The effects of timing of surgery on initial hospital course. Journal of Orthopaedic Trauma. Jul 2013;27(7):405-412. [PubMed: 23287766]

28.

Hannan EL, Zhong Y, Lahey SJ, et al. 30-day readmissions after coronary artery bypass graft surgery in new york state. Jacc: Cardiovascular Interventions. 2011;4(5):569-576. [PubMed: 21596331]

29.

Hannan EL, Racz MJ, Walford G, et al. Predictors of readmission for complications of coronary artery bypass graft surgery. JAMA. 2003;290(6):773-780. [PubMed: 12915430]

30.

Parina RP, Chang DC, Rose JA, Talamini MA. Is a low readmission rate indicative of a good hospital? Journal of the American College of Surgeons. 2015;220(2):169-176. [PubMed: 25529903]

31.

Stefan MS, Pekow PS, Nsa W, et al. Hospital performance measures and 30-day readmission rates. Journal of General Internal Medicine. 2013;28(3):377-385. [PMC free article: PMC3579957] [PubMed: 23070655]

32.

Thomas JW. Does risk-adjusted readmission rate provide valid information on hospital quality? Inquiry. Fall 1996;33(3):258-270. [PubMed: 8883460]

33.

Tsai TC, Joynt KE, Orav EJ, Gawande AA, Jha AK. Variation in surgical-readmission rates and quality of hospital care. New England Journal of Medicine. Sep 19 2013;369(12):1134-1142. [PMC free article: PMC4107655] [PubMed: 24047062]

34.

Zitser-Gurevich Y, Simchen E, Galai N, Braun D. Prediction of readmissions after cabg using detailed follow-up data: The israeli cabg study (iscab). Medical Care. 1999;37(7):625-636. [PubMed: 10424633]

35.

Auerbach AD, Hilton JF, Maselli J, Pekow PS, Rothberg MB, Lindenauer PK. Follow the crowd or shop for the best? Volume, quality, and outcomes of coronary artery bypass surgery. Annals of Internal Medicine. 2009;150(10):696-704. [PMC free article: PMC4617614] [PubMed: 19451576]

36.

Bhattacharyya T, Freiberg AA, Mehta P, Katz JN, Ferris T. Measuring the report card: The validity of pay-for-performance metrics in orthopedic surgery. Health Aff (Millwood). Mar-Apr 2009;28(2):526-532. [PMC free article: PMC3004748] [PubMed: 19276012]

37.

Brinkman W, Herbert MA, O'Brien S, et al. Preoperative beta-blocker use in coronary artery bypass grafting surgery: National database analysis. JAMA Internal Medicine. Aug 2014;174(8):1320-1327. [PubMed: 24934977]

38.

Cotogni P, Barbero C, Passera R, Fossati L, Olivero G, Rinaldi M. Violation of prophylactic vancomycin administration timing is a potential risk factor for rate of surgical site infections in cardiac surgery patients: A prospective cohort study. BMC cardiovascular disorders. 2017;17(1). [PMC free article: PMC5341467] [PubMed: 28270114]

39.

Kim ES, Park SW, Lee CS, Gyung Kwak Y, Moon C, Kim BN. Impact of a national hospital evaluation program using clinical performance indicators on the use of surgical antibiotic prophylaxis in korea. International Journal of Infectious Diseases. Mar 2012;16(3):e187-192. [PubMed: 22245171]

40.

Kurlansky PA, Argenziano M, Dunton R, et al. Quality, not volume, determines outcome of coronary artery bypass surgery in a university-based community hospital network. The Journal of Thoracic and Cardiovascular Surgery. 2012;143(2):287-293. e281. [PubMed: 22104669]

41.

LaPar DJ, Isbell JM, Kern JA, Ailawadi G, Kron IL. Surgical care improvement project measure for postoperative glucose control should not be used as a measure of quality after cardiac surgery. Journal of Thoracic & Cardiovascular Surgery. Mar 2014;147(3):1041-1048. [PubMed: 24418668]

42.

McDonnell ME, Alexanian SM, Junqueira A, Cabral H, Lazar HL. Relevance of the surgical care improvement project on glycemic control in patients undergoing cardiac surgery who receive continuous insulin infusions. Journal of Thoracic & Cardiovascular Surgery. Feb 2013;145(2):590-594; discussion 595-597. [PubMed: 23062416]

43.

Rasouli MR, Jaberi MM, Hozack WJ, Parvizi J, Rothman RH. Surgical care improvement project (scip): Has its mission succeeded? Journal of Arthroplasty. Aug 2013;28(7):1072-1075. [PubMed: 23602416]

44.

Schelenz S, Tucker D, Georgeu C, et al. Significant reduction of endemic mrsa acquisition and infection in cardiothoracic patients by means of an enhanced targeted infection control programme. Journal of Hospital Infection. Jun 2005;60(2):104-110. [PubMed: 15866007]

45.

Wang Z, Chen F, Ward M, Bhattacharyya T. Compliance with surgical care improvement project measures and hospital-associated infections following hip arthroplasty. Journal of Bone & Joint Surgery - American Volume. Aug 01 2012;94(15):1359-1366. [PMC free article: PMC3401140] [PubMed: 22740029]

46.

Guru V, Tu JV, Etchells E, et al. Relationship between preventability of death after coronary artery bypass graft surgery and all-cause risk-adjusted mortality rates. Circulation. 2008;117(23):2969-2976. [PubMed: 18541752]

47.

Smith T, Li X, Nylander W, Gunnar W. Thirty-day postoperative mortality risk estimates and 1-year survival in veterans health administration surgery patients. JAMA Surgery. 2016;151(5):417-422. [PubMed: 26747331]