Identifying surgical adverse events using ICD-9-CM codes

International Classification of Diseases, 9^th revision, Clinical Modification (ICD-9-CM or ICD) codes have been evaluated for a variety of purposes beyond documenting health conditions and procedures to support hospital billing. Uses have included characterizing the anatomical location of injury, ¹ injury severity, ^{2, 3} disease severity, ⁴ comorbidity severity, ⁵ the likelihood of death, ⁶ and the presence of medically caused injury. ^7–11 The resulting severity scores, in turn, have largely been used to predict resource use ¹² and to control for patient mix. ¹³

ICD-coded hospital discharge data have also been used to assess quality of care. They are easily accessed through electronic databases and tend to be available in statewide repositories. Research on the use of hospital administrative data for evaluating quality of care has produced mixed results, depending on the purpose of the researcher. Efforts have been made to identify complications, complications due to medical care, complications due to errors in medical care, complications resulting in harm, preventable complications, process errors, and near misses.

Studies of the validity of using ICD codes to identify adverse events (AEs) due to medical care have frequently involved the Complications Screening Program (CSP). ICD codes for identifying surgical AEs have been found to have some validity, while ICD codes identifying AEs in medical patients have been less successful. ^{7, 10, 11, 14} The diagnosis and external-cause-of-injury codes (E-codes) specifically designated for recording misadventures and complications of surgical and medical care (996–999, E870–E879, E930–E949) have also been used to study the trends and patterns of AEs. ¹⁵ Building on the CSP research, the Agency for Healthcare Research and Quality (AHRQ) recently introduced the Patient Safety Indicators (PSIs). ¹⁶ The PSIs are primarily based on ICD codes for specific patient populations. They are intended to flag patient safety problems and point to areas for investigation and intervention. Zhan and Miller found they could be used to identify patients who are more likely to die, have longer lengths of stay, and have higher charges. ⁹

In the study reported here, ICD codes representing diagnoses, procedures, and E-codes were evaluated for their ability to detect harm resulting from care management of surgical patients in acute care hospitals. No attempt was made to assess preventability, medical negligence, or error. The emphasis was on determining whether groups of ICD codes could be used for statewide surveillance of patient harm resulting from care during hospital stays.

This study is the result of a joint project between the Utah Department of Health (UDOH) and the Missouri Department of Health and Senior Services (MDHSS) under a cooperative agreement with AHRQ. The collaborative study addressed the accuracy and efficacy of using ICD codes, available in the mandated discharge data collection systems in both States, for surveillance and reporting on categories or classes of adverse events. UDOH evaluated ICD codes selected for their potential to identify adverse drug events (ADEs). For MDHSS, the focus was on AEs related to surgical hospital stays.

ICD-9-CM codes suggestive of surgical adverse events

An adverse event was considered to be an undesirable and unintended injury resulting from medical care management (an act of care provided by the hospital or the omission of necessary care, rather than from a patient's underlying disease process.) Harm was defined as death, prolonged hospital stay, or temporary or permanent impairment of bodily function or structure that requires intervention such as a change in monitoring of the patient's condition, a change in therapy, or active medical or surgical treatment or attention (if feasible or possible). Both harm and the degree to which the AEs were due to care management were operationalized by rating scales (Appendixes 2 and 3^**) in the data collection tool. Surgery AEs were defined as harm events related to the preoperative care, the surgery, or the postoperative care.

To assist with refining the ICD-9-CM classification of AEs, the Utah/Missouri project team convened a national expert panel comprised of physicians, medical record coders, pharmacy directors, and health care researchers. An initial set of 1,192 ICD codes were selected for the study based on the UDOH report, Adverse Events Related to Medical Care, Utah: 1995-99, a review of the literature, and other research in progress on the use of ICD codes as flags of in-hospital AEs. Following an orientation session and consensus building on the definition of AEs for this study, each panelist reviewed a worksheet including a subset of the codes with a description of each code. Each code was rated by a minimum of three and a maximum of nine panelists along three scales: medical care/causality, patient harm, and preventability. A total of 1,003 were retained for the study. With consultation from some expert panelists, the UDOH medical epidemiologists grouped the final list of codes into 66 classes. Codes selected as flags for surgery-related AEs numbered 377 and made up 23 adverse event classes (Appendix 1). ¹⁷ Additional details on the Utah/Missouri ICD-9-CM classification of adverse events, including the national panel exercise, will be addressed in a separate paper.

Sample design

Hospital discharge records for the first three quarters of 2001 formed the sampling frame for the study. Missouri hospitals and ambulatory surgery centers have been required by law to submit records to the MDHSS since 1993. Included on the records are patient identifiers, demographic information, up to 23 diagnoses and 20 procedures, type of admission, discharge disposition, and E-codes. In 2001 there were 123 general acute care hospitals accounting for 874,836 inpatient discharges.

A convenience sample of 40 hospitals was initially selected to provide data for this study and a second study designed to take advantage of the hospital license renewal surveys. Three of the 40 hospitals elected not to participate in the project, and a fourth was dropped due to scheduling problems with the licensure survey. The remaining 36 hospitals comprised the sampling frame for both studies. They tended to be somewhat larger and more urban than Missouri acute care hospitals as a whole. Forty-two percent had over 7,000 discharges (medical and surgical), compared to 32 percent of all 123 hospitals. Fifty-six percent of the 36 study hospitals were in Metropolitan Statistical Areas, compared to 49 percent of hospitals statewide.

Given the large number of codes selected for the AE classes, as well as constraints on the number of medical records that could be abstracted for this study, the sample design focused on representing the AE classes rather than individual codes or hospitals. It was initially estimated that approximately 1,000 surgical discharge records would allow a reasonable trade-off between an adequate statistical analysis of the 23 classes and a cost-manageable number of chart reviews. Records were sampled using the following criteria: First, only records with a surgical Diagnostic Related Group (DRG) were included. ¹⁸ Second, at least one of the flag codes had to be present in a secondary diagnosis field on the discharge record. (The principal diagnosis was assumed to represent a health condition existing at the time of admission and was not searched for flag codes.) Finally, the record was associated with a specific AE class based on the first flag code encountered when searching through the 22 secondary diagnosis and 20 procedure fields on the discharge record.

The sampling design was a doubly stratified random sample of the 23 AE classes. Each class was assigned to contribute a maximum of 30, 45, or 60 records, based on the number of records available for a class. Within each class, the hospitals were ranked according to the number of records that contained codes for that class. The hospitals were then divided among three roughly equal terciles—the top tercile containing hospitals with the most records for that class, the middle containing hospitals with a medium number of records, and the bottom tercile containing hospitals with the fewest records for the class. A hospital could be assigned to the top tercile in one class and the middle or bottom tercile in a different class. Each tercile was sampled proportionately to the percentage of records it contained for a class. Within these constraints, the records were selected randomly and without replacement. Hospitals having no records with a flag code in a given class were not sampled for that class. This double stratification design attempted to maximize the possibility that all classes and all strata, but not all hospitals, contributed records.

Due to chance factors, four hospitals did not contribute records, leaving 32 hospitals and 991 discharges in the final sample. Some attrition occurred during data collection due to factors such as unavailable charts, inadequate medical record documentation for the data abstraction, and errors in the specification of the record as an acute care discharge. At the conclusion of the chart review, 941 records representing 150 flag codes were available for analysis of the surgery AE classes.

Identifying surgical adverse events using medical record abstraction

The data collection tool was a computerized form adapted from Woloshynowych et al. ¹⁹ The tool was preloaded with all diagnosis codes, E-codes, procedure codes, and other items taken from the selected discharge record. Abstractors recorded whether the identified injury or complication was an AE, whether it originated prior to or after the current admission, and whether it related to surgery. After documenting the AE, abstractors recorded that the AE was associated with one of the preloaded (billed) ICD codes or with an ICD code not contained in the discharge record, or that the AE was not adequately described by any existing ICD code. The form provided for documenting up to three confirmed adverse events and recording the relationship, if any, between the AEs and the preloaded diagnosis codes, E-codes, or procedure codes. To facilitate review, the chart abstraction tool contained the study definitions of adverse event and harm, and required the reviewer to use scales to rate the level of harm (Appendix 2) and the degree to which the AE was due to care management (Appendix 3).

The medical chart review activity was conducted by Primaris, the Medicare quality improvement organization for the State of Missouri. Nurses experienced in medical record review were recruited to conduct the chart abstractions for the study. Qualifications of the nurse reviewers included 3 years of experience with clinical data abstraction in combination with either acute care or clinical coding experience.

The nurse reviewers were provided 40 hours of specialized training, focusing on adverse event identification, study definitions, record abstraction, ICD clinical coding, data collection tool completion, and data entry. A registered health information administrator provided the training on medical record coding and coding of adverse events. Special emphasis was placed on how and when to use E-codes.

Each nurse reviewer participated in a gold-standard testing process prior to field assignment for the project chart abstractions. Gold-standard test cases included one case in which no AE was documented and four cases that included one or more events that were consistent with the study definition of an AE. Only reviewers who passed the gold-standard test were retained for the project.

Comparison with other studies

Three studies have utilized the CSP to investigate the ability of groups of ICD codes to flag postadmission adverse events and substandard care in a large Medicare sample. ^{10, 11, 14} The CSP uses trigger codes to assign patients to risk pools (major surgery, minor surgery, etc.) and to one of 28 complications (reopening of surgical site, postoperative acute myocardial infarction, etc.). A computer algorithm examines the diagnoses, procedures, and procedure dates to determine whether the record meets the criteria for a preventable, postadmission complication due to substandard care. Lawthers et al. ¹⁰ studied the extent to which the CSP trigger codes were an accurate representation of the charts and whether they could identify patients with AEs that occurred after admission. Across 17 surgical complications, they found 73 percent confirmed as postadmission AEs. The rate for six medical complications was 32 percent. In a second study in this series, Weingart et al. ¹⁴ examined how well the CSP identified complications of care and potential quality-of-care problems. They found a PVP rate of 68 percent for 15 surgical complications and 27 percent for 5 medical complications. Thirty-six percent of the flagged patients were found to have quality-of-care problems. In the third study, McCarthy et al. ¹¹ looked at the extent to which complications identified in the CSP were supported by objective clinical evidence in the charts. For the 11 surgical complications studied, they found objective evidence in 69 percent of the charts. An additional 12 percent had only a physician's note, and 19 percent had neither. The medical complications had much lower confirmation rates.

Of these three studies, ours is most similar to the Weingart study. We looked at the ability of classes of ICD codes to flag patients who had AEs that were due to care management of the surgery patient during the current admission. We did not attempt to determine whether the care was substandard, as was done in the Weingart study. ¹⁴ For 15 surgical complications, Weingart et al. found a postadmission rate PVP of 68 percent and a substandard-care rate of 36 percent. Our postadmission PVP for surgery AEs due to care management across 23 classes was 37 percent. However, our sample was not restricted to Medicare patients, a population that is likely more at risk for adverse events. Also, our study was exploratory in nature and we did not expect all the classes and codes to have high PVPs.

Lessons learned

Though an extensive number of codes were studied, only a few classes and individual ICD codes performed well enough to be used to flag surgery AEs in a statewide surveillance system. Adding diagnosis dates or readmission codes to hospital discharge records would only modestly improve the usefulness of this group of ICD codes for surveillance of AEs, but would help to eliminate false positives. Greater improvement for the codes studied here would result from adding a marker to indicate that a certain diagnosis was the result of care management; however, there is no reason to believe such markers would be reported more reliably than are the E-codes for misadventures. Further study of which codes best identify which types of AEs in which populations, as is done with the PSIs, would be useful as we continue to determine how best to develop statewide surveillance of adverse events.

Limitations

Our results were not weighted to reflect the sampling scheme due to its complexity. Weighting might change the PVP percents somewhat, but probably not the relative performance of the classes. We also did not compute an interrater reliability coefficient. These have typically been modest in studies of this kind. ^21–23 The reviewers in this study were not physicians, but rather nurses who were carefully trained and tested prior to the study. While neither nurses nor physicians typically have expertise in clinical coding, it is possible that physician reviewers would have produced somewhat different results. A random sample of hospitals would have allowed firmer generalizations to the population than does our convenience sample, which slightly favored large, urban hospitals. Smaller hospitals do not have many surgical patients, however, so our results should not be overly unrepresentative in that respect.

Future directions

In order to build a more reliable ICD-based surveillance system for identifying adverse events, the second study in this series will focus on gaining a better understanding of which codes to include or exclude from the AE classes. Our study found inherent weaknesses in the validity of some of the defined AE classes, but also stand-alone strengths in particular ICD codes. Inspection of the classes with high PVPs found that the scores were driven by high PVPs of only one or two ICD codes in each class. Alternatively, some low-PVP classes appeared to contain one or two promising codes whose importance for surveillance purposes may have been masked by their low frequency in the sample.

Based on the results of this study, the followup study will focus more on the performance of individual codes within the AE classes. This strategy will facilitate the evaluation of the classes and the individual ICD codes for surveilling patient injury. Three classes (Heart Disease; Acute GI Ulcer, GI Bleeding, other GI Disorders; Disorders of Urinary System) have been dropped completely from the followup study. These classes did have a few codes specific to surgical complications, but the codes occurred very infrequently in the discharge population and would not be very useful for surveillance. These three classes also were among the more poorly performing classes, with PVPs less than 35 percent (Table 1).

Several of the ICD codes occurred with low frequency in our sample but are fairly frequent in the discharge population. Our followup study will attempt to obtain more reliable estimates of the validity of these codes by oversampling them. Conversely, those flag codes with adequate representation in this study will not be included in the followup study. This method should help us to identify the best classes and codes and to assess the overall value of the classes for statewide surveillance of surgery AEs.