Identifying Candidate Indicators

In the first phase of development, the UCSF-Stanford EPC conducted interviews with individuals affiliated with hospital associations, business coalitions, State data groups, Federal agencies, and academia about topics related to quality measurement. The interviews provided background information on measure use, suggested new indicators for potential development, and provided the names of additional individuals within the field who could be contacted for an interview. The interviews also suggested new risk-adjustment methods and assisted in framing the evaluation of potential indicators. With this information and relevant literature, the team developed a framework in which to evaluate the performance of the candidate measure. Table 2 provides an overview of the criteria used to evaluate the potential measures as well as a brief description of each.

Table 2

Criteria used to evaluate potential Quality Indicators

The research team also undertook a literature review that was structured in two phases. The first phase identified potential measures within the literature that were applicable to comparisons among providers or among geographic areas. In addition, potential indicators were identified using the various established databases of measures such as those from the Joint Commission for the Accreditation of Healthcare Organizations, Healthy People 2010, and so on. In the second phase of the literature review, the team performed an initial screen of the candidate indicators for relevance and accuracy. If an indicator met the criteria as described in Table 2, it received a comprehensive literature review and empirical evaluation.

The next phase of development was to identify potential risk-adjustment models for each of the selected candidate measures. Users of the QIs preferred a risk-adjustment system that was (1) open with published logic; 2) cost effective with data collection costs minimized and with any additional data collection being well justified; (3) designed using a multiple-use coding system, such as those used for reimbursement; and (4) officially recognized by government, hospital groups, or other organizations. In general, the All Patient Refined-Diagnosis Related Groups (APR-DRGs) tended to fit more of the user preferences than other alternatives considered. In addition, the APR-DRGs were reported to perform as well as or better than other risk-adjustment systems for several conditions.^7–9 The APR-DRGs are used in various AHRQ QIs; however, this method is not used with the PDIs, which use a novel and specialized risk-adjustment system that includes the data element Present on Admission (POA), the AHRQ Clinical Classification System, and stratification.

Analyzing Potential Indicators

The next step in the development process was empirical testing of all potential indicators. The primary datasets used were the HCUP Nationwide Inpatient Sample (NIS) and the State Inpatient Database (SID). The NIS is the largest all-payer inpatient care database in the United States, consisting of approximately 8 million hospital stays per year, specifically it consists of discharges of about a 20 percent stratified sample of community hospitals in the country. The SID consists of the universe of inpatient discharge abstracts in participating States, translated into a uniform format. This database encompasses about 90 percent of all community hospital discharges in the nation. More recently, the Kids’ Inpatient Database was used to develop the AHRQ PDIs. This database, currently the only all-payer inpatient care database for children in the United States, contains 2–3 million hospital discharges. For more information about these databases, please go to the AHRQ website at www.ahrq.gov/data/hcup.

The data from these databases were used to test each evaluation criterion that was assessed empirically i.e., precision, bias, and construct validity. The results of the candidate indicators were compared, and those indicators that performed poorly were eliminated. Bias tests were conducted to determine the need for risk adjustment, and then finally, construct validity was evaluated to provide evidence of the nature of the relationship between potential indicators.

The next phase of indicator development used multi-disciplinary clinician panel reviews. The team solicited nominations from professional clinical organizations and hospital associations, that were selected based on the applicability of the specialty or subspecialty to the candidate indicators. Nominees were chosen based on meeting certain criteria. For example, nominees were required to spend at least 30 percent of their work time on patient care, including hospitalized patients. The panelists were selected so that each group had a diverse membership in terms of clinical practice characteristics and settings.

The members of the panel were given a number of documents to evaluate the candidate measures. The documents provided included information about administrative data; coding from the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM); assignment of DRGs and Major Diagnostic Categories (MDCs); and specific definitions for adverse events or complications, preventability, and medical error. Candidate measure information incorporated exclusion and inclusion criteria, the clinical rationale for the indicator, and the specification criteria. A summary of literature-based evidence and empirical rates based on the NIS were provided for reference as well. Finally the panelists were given a list of potential questions regarding indicator definitions that the team planned to explore. Each panelist completed a 10-item questionnaire that asked them to determine the candidate indicator’s ability to screen out conditions present on admission, to identify conditions with high potential for preventability, to identify medical errors, or to evaluate access to high-quality outpatient care. Panelist were also asked to consider potential sources of bias, reporting or charting problems, potential ways of gaming the indicator, and possible adverse effects of implementing the measure. Finally, panelists were invited to suggest changes to the candidate indicator.

After the questionnaires were returned, the team convened a series of conference calls with the panelists to discuss their opinions regarding the candidate measures. Using a modified version of the RAND/UCLA method developed in the 1980s. The RAND/UCLA Appropriateness Method¹⁰ is used to synthesize the best available scientific evidence and expert opinion on health care issues. This method is a way to reach formal agreement on how the current science is interpreted by care givers in the real world. For the development of the QIs, the primary goal of the interaction was to allow for and encourage varied opinions about the appropriateness of an indicator. For our purposes, consensus was not the goal of the discussion, and agreement and disagreement on every indicator under consideration was noted. Following each conference call, modifications were made to each indicator as suggested by the panelists. The revised indicators were then redistributed to the panelists, along with questionnaires, and instructions to reevaluate and again rate each indicator based on their current opinion after the conference call discussions. Once the final round of questionnaires was received, the team calculated median scores to determine the degree of agreement among panelists. In addition, the team calculated scores indicating the level of acceptability of the indicator and the dispersion of ratings across the panel. The following criteria covered in the questionnaire were used to summarize the panel’s options on each indicator:

• Overall usefulness of the indicator, both for internal quality improvement purposes and comparisons between hospitals
• Likelihood that the indicator measures a complication and not a comorbidity (specifically, present on admission)
• Preventability of complication
• Extent to which a complication is due to medical error
• Likelihood that a complication that occurs is charted
• Extent that the indicator is subject to bias (systematic differences, such as case mix, that could affect the indicator in a way not related to quality of care)

For area-level indicators, panelists provided feedback on the following areas:

• Overall usefulness of the indicator, both internally within an area and for comparisons between areas
• Extent to which an event reflects poor access to quality outpatient care
• Consistency in terminology for charting the principal diagnosis
• Extent that the indicator is subject to bias

The next step in the development process involved peer review of the candidate measures. Nominations were sought for clinicians, policy advisors, professors, researchers, and managers in quality improvement to participate on this panel. The group was instructed to provide comments on the indicators with constructive suggestions for content and presentation enhancements.

Once the panel reviews and evaluations were complete, the candidate indicators could go through further empirical testing to refine their definitions. After that, the indicator may undergo further clinical and peer review, which can occur over several rounds until the definition of the indicator is finalized. As with any measure of performance, the process of refinement is ongoing and becomes part of the measure maintenance activities of the measure developer. Figure 1 provides a graphic account of the basic development process.

Figure 1

The AHRQ Quality Indicator development and evaluation process

As a measure developer, AHRQ maintains these measures and on an annual basis, provides revisions to the measures, including ICD-9-CM and DRG code updates, an update to the reference population used in calculating the QIs, and refinement of the specifications based on additional evidence in the literature and user input. Literature reviews are completed on one QI module every year, which allows time for new research to be completed and subsequently published in peer reviewed journals.

The AHRQ QIs and Their Data Sources

There are several sources of data that can be used to measure performance, and these data sources can be grouped into the following categories: administrative data (also known as billing or claims data), medical record information, patient-derived data (i.e., surveys);¹¹ confidential reports from providers, and direct observation. All categories of data have strengths and weaknesses and each data source should be evaluated for comprehensiveness or the completeness of data elements as they pertain to individuals, and inclusiveness or the extent to which populations are represented in a particular geographic area.¹² The AHRQ QIs use data derived from administrative databases, which is considered a “by-product” of care delivery, i.e., reimbursement to hospitals or physicians or determining insurance eligibility of patients.⁹

While administrative data were not originally intended to be used in research, these types of databases are often used by researchers in their studies and clearly offer some important advantages, such as the ability to track study subjects over time. Administrative data are also relatively inexpensive to collect and readily available to researchers, administrators, and others. Additional advantages include the large sample sizes associated with this type of data, the ease of collection without interference with the care of the patient, the population-based characteristic of administrative data, and identifiers associated with the data that permit observations across sites and settings of care. The AHRQ QIs can be used with any administrative data set and largely rely on the ICD-9-CM codes for diagnosis and procedures from individual hospitalization data, which are derived from the 2004 Uniform Bill (UB-04). Other information such as patient identifiers, hospitalization descriptors, admission types, insurance information, and charge data can be found on this form. Thus by putting together the range of ICD-9-CM codes and supplementary codes such as E codes^* and V codes,^† and from a creative and clinically informed use of these codes, a picture of a patient’s clinical status and risk factors begins to form (see Table 3).

Table 3

Characteristics and Uses of Hospital Administrative Data

The AHRQ QIs are valuable because they are based on widely available data that can be used to assess quality. Theses QI indicators also have uniform definitions and standardized algorithms that can be used with virtually any administrative data set, which allows for comparisons across States, regions, communities, and hospitals.

As with any data source used to assess performance, there are a number of drawbacks to using administrative data to examine the quality of care delivered by health care providers. Despite the large number of ICD-9-CM codes available and the implied detail they contain, these codes do not have operational clinical definitions assigned, which make assignment by coders somewhat variable. While coders are generally formally trained in coding methods and instructed to use the terminology in the medical record, clinicians seldom use a consistent lexicon in their charting. Thus, the meaning of codes without a clinical context, or without the considerations of disease progression, and the interaction of comorbidities can provide an inaccurate clinical picture—limiting the usefulness of the data. Yet despite this limitation, data availability, coding systems, and coding practices are improving, which enhance our ability to identify quality problems as well as success stories, which can be further identified and studied.

Prevention Quality Indicators (PQIs)

The AHRQ PQIs are one set of quality measures that can be used to identify potential problems; follow trends over time; and ascertain disparities across regions, communities, and providers. This module focuses on preventive care services—outpatient services that assist individuals with either staying healthy or managing chronic illness. In these instances, inpatient data can provide information on admissions for ambulatory sensitive conditions that evidence suggests could have been avoided, at least in part through better outpatient care. For example, patients with diabetes may be hospitalized due to complications for their disease if their conditions have not been adequately monitored or if they do not receive education that would allow them to self-manage the disease. There are currently 14 PQIs, listed in Table 4 that measure rates of admissions to the hospital.

Table 4

AHRQ Prevention Quality Indicators

Factors such as poor environmental conditions or lack of patient adherence to treatment regimes can result in hospitalization. However, the PQIs provide a good starting point to assess quality of services within a community. The POIs can be used to provide a picture of health care in the community by identifying unmet needs, monitoring how well complications are being avoided in the outpatient setting, assessing access to health care, and comparing the performance of local health care systems across communities.

The PQIs represent the current state of the art in assessing the health care system as a whole, but particularly in the area of ambulatory care, for example, in preventing medical complications for both acute illness and chronic conditions. The PQIs are valuable when calculated at the population or area level and when used by organizations such as public health groups, State data organizations, health plans, large health systems, and other organizations concerned with the health of populations. The PQIs are risk adjusted for age and gender and provide information about potential problems in the community that may require further analysis. The PQIs help answer questions such as

• Does the admission rate for diabetes complications in my community suggest a problem in the provision of appropriate outpatient care to this population?
• How does the admission rate for congestive heart failure vary over time and from one region of the country to another?

These are just a few of the questions that the PQIs can address to assist those health care providers with responsibility for the health of a particular population. The PQIs allow for comparisons across States, regions, and local communities over time. The PQIs do not measure hospital quality, but reflect the care provided in the community.

Despite their strengths, there are several considerations when using these indicators. Differences in PQI rates can explain some of the variation across areas but not all. The complexity of the relationship between socioeconomic status and PQI rates makes it difficult to delineate how much of the observed relationships are due to true access to care issues, difficulties in potentially underserved populations, or other patient characteristics unrelated to quality of care that vary systematically by socioeconomic status. Second, the evidence related to potentially avoidable hospital admissions is limited for each indicator because many of the indicators have been developed as parts of sets. Finally, despite the relationships demonstrated at the patient level between higher quality ambulatory care and lower rates of hospital admission, few studies have directly addressed the question of whether effective treatments in outpatient settings would reduce the overall incidence of hospitalizations.

The Inpatient Quality Indicators (IQIs)

The AHRQ IQIs provide information about the quality of medical care delivered in a hospital. This measure set represents the state of the art in measuring the quality of hospital care using inpatient administrative data. The IQIs include measures in the areas of inpatient mortality; utilization of procedures for which there are questions of overuse, underuse, or misuse; and volume of procedures for which there is evidence that a higher volume is associated with lower mortality.

The IQIs that focus on volume are proxy measures of quality and represent counts of admissions in which these procedures were performed. They are based on evidence suggesting that hospitals that perform more of certain procedures—for example, those that are intensive, high-technology, or highly complex—may have better outcomes for those procedures. The provider-level volume IQIs are:

• Esophageal resection volume
• Pancreatic resection volume
• Abdominal aortic aneurysm (AAA) repair volume
• Coronary artery bypass graft (CABG) volume
• Percutaneous transluminal coronary angioplasty (PTCA) volume
• Carotid endarterectomy (CEA) volume

The mortality indicators for inpatient procedures cover procedures for which mortality has been shown to vary across institutions and for which there is evidence that high mortality may be associated with poorer quality of care. The mortality indicators for inpatient surgical procedures are:

• Esophageal resection mortality rate
• Pancreatic resection mortality rate
• AAA repair mortality rate
• CABG mortality rate
• PTCA mortality rate
• CEA mortality rate
• Craniotomy mortality rate
• Hip replacement mortality rate

When evaluating mortality rates, the corresponding volumes should be examined in conjunction with the mortality rate because that provides more information about the care delivered. For example, esophageal resection is a complex surgery, and studies have noted that providers with higher volumes have lower mortality rates. These results suggest that providers with higher volumes have some characteristics, either structurally or with regard to processes that influence mortality.

The mortality indicators for inpatient conditions cover conditions for which mortality has been shown to vary substantially across institutions and for which evidence suggests that high mortality may be associated with deficiencies in the quality of care. The mortality indicators for inpatient medical conditions are:

• Acute myocardial infarction (AMI) mortality rate
• AMI mortality rate, without transfer cases
• Congestive heart failure mortality rate
• Acute stroke mortality rate
• Gastrointestinal hemorrhage mortality rate
• Hip fracture mortality rate
• Pneumonia mortality rate

Also included in the IQIs are utilization indicators that examine procedures whose use varies significantly across hospitals and for which questions have been raised about overuse, underuse, or misuse. High or low rates for these indicators are likely to represent inappropriate or inefficient delivery of care. The procedure utilization indicators are:

• Cesarean section delivery rate
• Primary cesarean delivery rate
• Vaginal birth after cesarean (VBAC) rate, all
• VBAC rate, uncomplicated
• Laparoscopic cholecystectomy rate
• Incidental appendectomy in the elderly rate
• Bilateral cardiac catheterization rate

There are currently 28 IQIs that are measured at the provider or hospital level, as well as 4 area-level indicators that are suited for use at the population or regional level. These 4 indicators, which are utilization measures, include:

• CABG area rate
• Hysterectomy area rate
• Laminectomy or spinal fusion area rate
• PTCA area rate

The IQIs can be used by a variety of stakeholders in the health care arena to improve quality of care at the level of individual hospitals, the community, the State, or the Nation. The IQIs represent advancement in assessing quality of care using hospital administrative data. While these data are relatively inexpensive and convenient to use and represent a rich data source that can provide valuable information, like other data sources that have various limitations, the data should be used carefully when assessing and interpreting the quality of health care within an institution.

The Patient Safety Indicators (PSIs)

The PSIs are a set of quality measures that use hospital inpatient discharge data to provide a perspective on patient safety.¹³ Specifically, the PSIs identify problems that patients experience through contact with the health care system and that are likely amenable to prevention by implementing system level changes. The problems identified are referred to as complications or adverse events. There are currently 27 PSIs that are defined on two levels: the provider level and the area level. They are risk adjusted using a model that incorporates DRGs (with and without complications aggregated); a modified comorbidity index based on a list developed by Elixhauser and colleagues;¹⁴ and age, sex, and age-sex interactions.

At the provider level, the PSIs present a picture of patient safety within a hospital and provide information about the potentially preventable complication for patients who received their initial care and experienced the complication of care within the same hospitalization. The PSIs use secondary diagnosis ICD-9-CM codes to detect complications and adverse events. The measure set covers a variety of areas such as selected postoperative complications, selected technical adverse events, technical difficulty with procedures, and obstetric trauma and birth trauma. The 20 provider-level PSIs include:

• Postoperative pulmonary embolism or deep vein thrombosis
• Postoperative respiratory failure
• Postoperative sepsis
• Postoperative physiologic and metabolic derangements
• Postoperative abdominopelvic wound dehiscence
• Postoperative hip fracture
• Postoperative hemorrhage or hematoma
• Decubitus ulcer
• Selected infections due to medical care
• Iatrogenic pneumothorax
• Accidental puncture or laceration
• Foreign body left in during procedure
• Birth trauma—injury to neonate
• Obstetric trauma—vaginal delivery with instrument
• Obstetric trauma—vaginal delivery without instrument
• Obstetric trauma—cesarean section delivery
• Complications of anesthesia
• Death in low-mortality DRGs
• Death among surgical inpatients with treatable serious complications (previously known as Failure to rescue)
• Transfusion reaction (AB/Rh)

The area-level PSIs capture all cases of the potentially preventable complications that occur in a given area (e.g., metropolitan service areas or counties), either during hospitalization or resulting in subsequent hospitalizations. They are specified to include the principal diagnosis, as well as secondary diagnoses, for the complications of care. The measurement specifications add cases where a patient’s risk of the complication occurred in a separate hospitalization. The seven area-level PSIs are:

• Foreign body left in during procedure
• Iatrogenic pneumothorax
• Selected infections due to medical care
• Postoperative wound dehiscence in abdominopelvic surgical patients
• Accidental puncture or laceration
• Transfusion reaction
• Postoperative hemorrhage or hematoma

Widespread consensus exists that health care organizations can reduce patient injuries by improving the environment for safety—from implementing technical changes, such as electronic medical record systems, to improving staff awareness of patient safety risks. Clinical process interventions also present strong evidence for reducing the risk of adverse events related to a patient’s exposure to hospital care. These PSIs can be used to better prioritize and evaluate local and national initiatives. Some potential actions, after an in-depth analysis of the system and process of care, include the following:

• Review and synthesize the evidence base and best practices from scientific literature.
• Work with the multiple disciplines and departments involved in care of surgical patients to redesign care based on best practices with an emphasis on coordination and collaboration.
• Evaluate information technology solutions.
• Implement performance measurements for improvement and accountability.

The ability to assess all patients at risk for a particular patient safety problem, along with the relative low cost of collecting the data, are particular strengths of the datasets that use administrative data. However, many important areas of interest, such as adverse drug events, cannot currently be monitored well using administrative data and using this data source to identify patient safety events tends to favor specific types of indicators. For example, the PSIs cited in this chapter contain a large proportion of surgical indicators, rather than medical or psychiatric measures, because medical or psychiatric complications are often difficult to distinguish from comorbidities that are present on admission. In addition, medical populations tend to be more heterogeneous than surgical populations, especially elective surgical populations, making it difficult to account for case mix.

While PSIs may be more applicable to patient safety when limited to elective surgical admissions, the careful use of administrative data holds promise to identify problems for further analysis and study. The limitations of this measure set include those inherent with the use of administrative data, clinical accuracy of the discharged-based diagnosis coding, and indicator discriminatory power. Specifically,

• Administrative data are unlikely to capture all cases of a complication, regardless of the preventability, without false positives and false negatives (sensitivity and specificity).
• When the codes are accurate in defining an event, the clinical vagueness inherent in the description of the code itself (e.g., hypotension) may lead to a highly heterogeneous pool of clinical states represented by that code.
• Incomplete reporting is an issue in the accuracy of any data source used for identifying patient safety problems, as medical providers might fear adverse consequences as a result of full disclosure in potentially public records such as discharge abstracts.
• The heterogeneity of clinical conditions included in some codes, lack of information about event timing available in these datasets, and limited clinical detail for risk adjustment all contribute to the difficulty in identifying complications that represent medical error or that may be at least in some part preventable. These factors may exist for other sources of patient safety data as well. For example, they have been raised in the context of the Joint Commission’s implementation of a sentinel event program geared to identifying serious adverse events that may be related to underlying safety problems.

Yet, despite these issues, the PSIs are a useful tool to identify areas in patient safety that need monitoring and/or intervention for improved patient care.

The Pediatric Quality Indicators (PDIs)

The AHRQ PDIs are a set of quality measures that use hospital administrative data and involve many of the same challenges associated with measure development for the adult population. These challenges include the need to carefully define indicators, establish validity and reliability, detect bias, design appropriate risk adjustment, and overcome challenges of implementation and use. However, as a special population, children require special tailoring of quality measures and risk-adjustment methodologies. The AHRQ PDIs, developed through careful, ongoing research efforts, provide a risk-adjusted tool to identify quality problems for hospitalized children as well as assess the rate of potentially preventable hospitalizations. The AHRQ PDIs currently consist of 18 indicators defined as both provider- and area-level measures. The 13 provider-level PDIs are:

• Accidental puncture and laceration
• Decubitus ulcer
• Foreign body left in during procedure
• Iatrogenic pneumothorax in neonates
• Iatrogenic pneumothorax in non-neonates
• Pediatric heart surgery mortality
• Pediatric heart surgery volume
• Postoperative hemorrhage or hematoma
• Postoperative respiratory failure
• Postoperative sepsis
• Postoperative wound dehiscence
• Selected infections due to medical care
• Transfusion reaction

Existing risk-adjustment strategies for pediatric patients were not suitable for use with the AHRQ PDIs. Most available schemes apply to specific clinical groups and utilize clinical data not available in administrative databases. The APR-DRG methodology, used for risk adjustment in the adult population, was considered for use in the pediatric population. However, using the APR-DRGs could not adjust for complications in the pediatric population because it resulted in over-adjustment. As a result, different risk adjustment strategies were investigated for potential incorporation into the PDIs. Three important risk-adjustment factors of significance to the pediatric population were identified: (1) reason for admission (including principal procedure), (2) comorbidities, and (3) age and gender. Using a modified-DRG risk adjustment combined with comorbidity adjustment based on the AHRQ Clinical Classification System and age and gender adjustment, the AHRQ PDIs include a novel and specialized risk-adjustment system. They also include stratification, another approach to accounting for case mix. Stratification allows hospitals to identify which segment of the pediatric population accounts for any elevation in rates, creating more user-friendly measures. Tailored stratification schemes are available for six of the PDIs: accidental puncture and laceration, decubitus ulcer, iatrogenic pneumothorax, postoperative hemorrhage or hematoma, postoperative sepsis, and selected infections due to medical care. Despite these efforts to account for risk, it is anticipated that further research on pediatric risk adjustment will be important for assessing quality appropriately.

In addition to the provider-level indicators, the PDIs also include five area-level indicators:

• Asthma admission rate
• Diabetes short-term complication rate
• Gastroenteritis admission rate
• Perforated appendix admission rate
• Urinary tract infection admission rate

These indicators track potentially preventable hospitalizations and allow policymakers to target specific groups that appear to be developing more severe disease requiring hospitalization. Higher-than-anticipated rates may reflect poor access to care (e.g., from lack of insurance or too few primary care physicians), barriers to timely care (e.g., clinics that require daytime appointments), barriers to adherence to medical advice (e.g., language barriers), cultural influences that preclude seeking early treatment, or higher prevalence of poor health behaviors (e.g., smoking). Interventions may address any of these factors.

Area-level indicators are prone to bias due to cultural factors that may be outside of a health system’s control. For instance, an area with a high number of illegal immigrants may have patients presenting with more advanced disease, because patients delay seeking care for fear of deportation. In addition, factors such as smoking or obesity may be more prevalent in certain areas. Risk adjustment should include these factors, and an adjustment for socioeconomic status, as a proxy, and has been included in these PDIs. However, risk adjustment for socioeconomic groups may mask true differences in access to good quality care. For this reason, risk-adjusted rates should be considered alongside raw, unadjusted rates.

Quality Improvement

Originally, the AHRQ QIs were designed as an internal quality improvement tool to assist hospitals to identify and target potential areas for interventions. The ability to track quality of care for a wide range of patients is an important consideration for quality improvement. Hospitals, health care systems, and hospital associations use the AHRQ QIs for internal quality improvement, specifically to initiate case finding, root-cause analyses, and cluster identification, as well as to evaluate the impact of local interventions and to monitor performance over time.

Yet, as with any quality measures, these indicators must be used with care, because the administrative data on which the measures are based are not collected for research purposes or for measuring quality of care, but for billing purposes. While these data are relatively inexpensive to collect, convenient to use, and represent a rich source of information that provides valuable insights, they are one view of the multi-dimensional concept of quality. Our health care system currently uses a “hybrid” model derived from multiple sources, both electronic and paper records, to result in performance information. While not the only use of administrative data, it can as a sole data source, be used by individual hospitals to launch investigations into reasons for identified quality problems. Further study may:

• Reveal real quality problems for which quality improvement programs can be initiated.
• Uncover problems in data collection that can be remedied through stepped-up efforts to code more diligently.
• Determine that additional clinical information is required to understand the quality issues, beyond what can be obtained through billing data alone.

Overall, the AHRQ QIs are a valuable tool that takes advantage of readily available data to identify quality-of-care problems. Hospitals may use existing data to identify indicators with higher-than-expected rates, flagging potential quality concerns. These areas of concern may be investigated further to identify the underlying cause of the poorer-than-expected performance. In some cases, incorrect coding practices may be identified; in other cases, closer examination of system-level factors may be in order. Interventions may be devised to improve performance, and hospitals may track their own performance over time to identify areas for improvement.

Public Reporting and Pay for Performance

The AHRQ QIs are currently being used in several public reporting and pay for performance (P4P) initiatives at the national, State, and regional levels. At the national level, for example, they are used in tracking the quality of health care in the United States in the National Healthcare Quality Report¹⁶ and National Healthcare Disparities Report¹⁷ produced annually by AHRQ. These reports focus on four dimensions of quality—effectiveness, safety, timeliness, and patient centeredness—and are available on the AHRQ Web site. Other uses of the QIs include surveillance of trends over time at the State and community level as well as assistance in tracking disparities across areas, when the data are available.

Several organizations have incorporated the AHRQ QIs in reports on quality that allow for comparisons of individual hospitals. Organizations such as the Colorado Health and Hospital Association, the Texas Health Care Information Council, the Niagara Health Quality Coalition, and Norton Healthcare are a few. Many of these reports are Web based and are routinely updated. For a more complete list of organizations that use the AHRQ QIs for public reporting, see Table 6.

Table 6

Organizations Using the AHRQ QIs for Public Reporting

Organizations such as the Centers for Medicare & Medicaid Services and Anthem Blue Cross and Blue Shield of Virginia have incorporated selected AHRQ QIs into P4P demonstration projects or similar initiatives. These projects reward providers for superior performance based on a combination of performance measures, including the AHRQ QIs. Results from the Centers for Medicare & Medicaid Services demonstration project indicate that tying payment to performance may provide some incentive to improve the quality of care.

There are a number of factors to be considered when using the AHRQ QIs for public reporting and payment purposes. Factors related to data source and measurements raise important issues such as:

• Very low or low volume (small cell size) could impact patient confidentiality and also limit the ability to reliably identify quality differences.
• Measures may not be applicable to the majority of hospitals or applicable only to hospitals with specific services (e.g., cardiac surgery, obstetrics).
• Volume is a proxy measure; volume may be manipulated leading to concerns about appropriate utilization.
• Potential confounding bias or the impact may be impaired by skewed distribution not completely eliminated by risk adjustment or carefully constructed operational definitions.
• Benchmarking or the correct rate may not be clear.
• Many procedures are currently done on an outpatient basis or observation status.
• The indicator may require data not present in all administrative datasets, or risk adjustment may be inadequate when based only on data available from ICD-9-CM codes.
• Coding may vary across hospitals; some hospitals code more thoroughly than others, making fair comparisons across hospitals difficult.

However even with these limitations, codes, coding systems, and coding practices are improving and are often subject to auditing or monitoring for accuracy. Coders are becoming more aware of the importance of properly coding the data and how they are used in relation to quality improvement, public reporting, P4P and other initiatives.

Ideally, in public reporting and P4P initiatives, the results of the performance measures should be made available to those hospitals participating, along with information on averages for peer groups, for the State, and for the Nation.

It is important when using not only the AHRQ QIs but all measures used for purposes such as comparative reporting, purchasing, or payment to continually assess and evaluate them and provide feedback to the measure developer for measure refinement and improvement purposes. The process of measure development and maintenance is constant, and measure developer like AHRQ welcome input from uses in an effort to continue to refine and enhance the measures.

Research

A number of the AHRQ QIs have been used in health care research projects. On the whole, researchers use the indicators because of the quality and level of detail of the AHRQ documentation of the QIs as well as the fact that these measures capture important aspects of clinical care⁴⁰ (p. v). The AHRQ QIs, their documentation, and the related software reside in the public domain and are downloadable from the AHRQ Web site, free of charge. The QIs can be used with readily available administrative data, which researchers have ready access to in the form of HCUP. Further, researchers appreciate the fact that they can dissect indicator results and relate them back to individual records, which helps to gain a better understanding of the logic used in the measures, which, in turn, assists in distinguishing data quality issues from actual quality problems⁴⁰ (p. vii). Topics of studies using the AHRQ QIs include an analysis examining the association between the Joint Commission accreditation scores and the AHRQ IQIs and PSIs,⁴¹ the effect of resident physician work hour limits on surgical patient safety,⁴² and the determination of whether persons with Alzheimer’s disease were at greater risk for in-hospital mortality than non-Alzheimer’s patients.⁴³

Table 5, which is based on an environmental scan commissioned by AHRQ and completed by Hussey and colleagues,⁴⁰ indicates that the AHRQ QIs are frequently used by researchers in their projects.