Postmortem Examinations

Description of the data source Postmortem examinations, often referred to as autopsies, are highly specialized surgical procedures that are conducted to determine the cause of death or extent of disease. Hoyert (2011, p. 1) identifies two primary types of postmortem exams conducted in the United States: (1) “hospital or clinical autopsies, which family or physicians request to clarify cause of death or assess care,” and (2) “medicolegal autopsies, which legal officials order to further investigate the circumstances surrounding a death.” Postmortem exams may vary from an external-only exam to a full external and internal exam, depending on the request. While this chapter focuses on full-body postmortem exams, Chapter 6 describes the potential future state of postmortem examinations, which may include more minimally invasive approaches, such as medical imaging, laparoscopy, biopsy, histology, and cytology.

Notes about the data source Postmortem exams are considered a very strong method for identifying diagnostic errors because of the extensiveness of the examination that is possible (Graber, 2013; Shojania, 2002). However, there are some limitations to this data source for the purpose of estimating the incidence of diagnostic error. Postmortem exams are conducted on people who have died; thus, the results can only provide information about diagnostic errors that led to the patient's death and about other diseases present that had not been previously identified, whether or not they contributed to the patient's death. A very limited number of postmortem exams are performed annually, and postmortem exam rates can also vary geographically and institutionally. Little information is available for characterizing the relationship between those who receive postmortem exams and the potential number of eligible cases, but those who undergo autopsy are more likely to have experienced a diagnostic error and that error is more likely to have contributed to the patient's (premature) death (an example of selection bias) (Shojania, 2002).

Methods for identifying cases for review (denominator) The decision about whether an individual patient will receive a postmortem exam is based on requests from clinicians or family members as well as on local criteria set by coroners or medical examiners. With the exception of postmortem examinations done for criminal forensic purposes, family members must consent to having the procedure done. There is no systematic information on the frequency with which the request for an autopsy is refused (which would introduce response bias into results). The performance of postmortem exams has declined substantially in the United States in recent decades (Lundberg, 1998). National data on postmortem exams have not been collected since 1994; at that time, fewer than 6 percent of non-forensic deaths underwent a postmortem exam (Shojania et al., 2002).

Research studies that have used postmortem exam results have used consecutive series, prespecified criteria (including randomly selected autopsies), or convenience samples (Shojania, 2002).

Methods for determining if an error occurred (numerator) The results of the postmortem exam typically provide a cause of death and a description of the presence and severity of other diseases. These results are compared to another data source, typically medical records or interviews with treating clinicians or family members. Discrepancies between what was found in the postmortem exam and what was known prior to that are the basis for determining the occurrence of a diagnostic error. Such determinations are subject to the reliability and validity of both the postmortem exam findings and the results from the data collected from the original sources.

What is known Postmortem examinations have been described as an important method for detecting diagnostic errors (Berner and Graber, 2008; Graber, 2013). In their review of postmortem examination data, Shojania and colleagues concluded that “the autopsy continues to detect important errors in clinical diagnosis” (Shojania et al., 2002, p. 51). On average, 10 percent of postmortem exams were associated with diagnostic errors that might have affected patient outcomes (i.e., Class I errors).² They estimated that the prevalence of major errors (i.e., Class I and II errors) related to the principal diagnosis or the cause of death was 25 percent. Some incidental findings found during postmortem exams should not be classified as diagnostic errors; of primary importance is identifying diagnostic errors that contributed to a patient's death (Class I errors).³ Shojania and colleagues noted that some selection bias is reflected in this estimate because the cases in which there was more uncertainty about the diagnosis were more likely to undergo postmortem exam. A systematic review of diagnostic errors in the intensive care unit found that 8 percent of postmortem exams identified a Class I error and that 28 percent identified at least one diagnostic error (Winters et al., 2012). According to Shojania et al. (2003, p. 2849), the rates of autopsy-identified diagnostic errors have declined over time but remain “sufficiently high that encouraging ongoing use of the autopsy appears warranted.” Based on their findings, they estimated that among the 850,000 individuals who die in U.S. hospitals each year, approximately 8.4 percent (71,400 deaths) have a major diagnosis that remains undetected (Shojania et al., 2003).

Opportunities for improvement The committee concluded that postmortem exams play a critical role in understanding the epidemiology of diagnostic errors and that increasing the number of such exams is warranted. In addition, tracking the number of deaths, those eligible and selected for postmortem exams, and the refusal rate among family members would enable the development of better national estimates of diagnostic error incidence. The committee weighed the relative merits of increasing the number of postmortem examinations conducted throughout the United States versus a more targeted approach. The committee concluded that it would be more efficient to have a limited number of systems who are highly qualified in conducting postmortem exams participate to produce research-quality information about the incidence and nature of diagnostic errors among a representative sample of patient deaths. This approach reflects both financial realities and workforce challenges (i.e., a limited number of pathologists being available and willing to conduct a large number of such exams) (see also Chapter 6). The systems that are selected to routinely conduct postmortem exams could also investigate how new, minimally invasive postmortem approaches compare to full-body postmortem exams.

Medical Records

Description of the data source A medical record is defined as a documented account of a patient's examination and treatment that includes the patient's clinical history and symptoms, physical findings, the results of diagnostic testing, medications, and therapeutic procedures. The medical record can exist in either paper or electronic form.

Notes about the data source Medical records exist only for patients who have sought care from a clinician, team, or facility. Although there are some common conventions for structuring medical records (both in paper and electronic formats), much of the content of the record depends on what the clinician chooses to include; thus, there may be variations in the extent to which clinical reasoning is documented (e.g., what alternative diagnoses were considered, the rationale for ordering [or not ordering] certain tests, and the way in which the information was collected and integrated). Both regulatory and local rules affect which members of the diagnostic team contribute to the documentation in a medical record and how they contribute. Except in highly integrated systems, patients typically have a separate medical record associated with each clinician or facility from which they have sought care. When patients change their source of care, the information from medical records maintained by the previous clinicians may or may not be incorporated into the new record.

Methods for identifying cases for review (denominator) The most common methods for identifying cases for review are either to draw a random sample of records from a facility (especially hospitals), clinic, or clinician practice or to assemble a criteria-based sample (e.g., a trigger tool). The criteria-based tools typically select events that have been associated with a higher probability of identifying a diagnostic error, such as unplanned readmissions to a hospital, emergency department visits after an outpatient visit, or the failure of a visit to occur after an abnormal test result. Estimates of the incidence of diagnostic errors based on medical records need to account for the probability that an individual is included in the study sample and the likelihood that a visit (or set of visits) requires that a diagnosis be made. Because these factors likely vary by geography and patient populations, arriving at national estimates from studies done in limited geographic areas is difficult.

Methods for determining if an error occurred (numerator) There are two common methods for determining if an error occurred: implicit and explicit. In the implicit method, an expert reviewer, taking into account all of the information that is available in the medical record, determines whether or not an accurate or timely diagnosis was made and, if a defect in the process occurred, the nature of that problem. In the explicit method, specific criteria are developed and data are abstracted from the medical record to determine whether or not an error occurred. The reliability of implicit and explicit methods for assessing quality of care and patient safety has been studied. Generally, implicit methods have been found to be less reliable than explicit methods (Hofer et al., 2004; Kerr et al., 2007). In the Utah and Colorado Medical Practice Study, which was one of the sources for estimating medical errors in the IOM's To Err Is Human report, the inter-rater reliability (agreement among reviewers) was κ=0.40–0.41 (95 percent confidence interval, 0.30–0.51) for identifying adverse events and κ=0.19–0.24 (95 percent confidence interval, 0.05–0.37) for identifying negligent adverse events (Thomas et al., 2002). These rates are considered moderate to poor (Landis and Koch, 1977). The reliabilities for the Harvard Medical Practice Study were in the same range (Brennan et al., 1991). Zwaan et al. (2010) reported a reliability of κ=0.25 (95 percent confidence interval, 0.05–0.45) (fair) for identifying adverse events and of κ=0.40 (95 percent confidence interval, 0.07–0.73) (moderate) for whether the event was preventable. Reliability in turn can affect the event rate that is reported. By contrast, the inter-rater reliability for explicit review of records for quality studies has been reported at approximately 0.80 (McGlynn et al., 2003).

What is known Two studies based on medical record reviews reported in the literature in the 1990s and early 2000s estimated that diagnostic errors account for 7 and 17 percent of adverse events in hospitalized patients, respectively. In the Harvard Medical Practice Study of more than 30,000 patient records, diagnostic errors were identified in 17 percent of the adverse events (Leape et al., 1991). A review of 15,000 records from Colorado and Utah found that diagnostic errors constituted 6.9 percent of adverse events (Thomas et al., 2000).

More recently, Zwaan and colleagues conducted a retrospective patient record review to assess the occurrence of diagnostic adverse events (harm associated with a diagnostic error) within hospitals in the Netherlands (Zwaan et al., 2010). Those researchers found that diagnostic adverse events occurred in 0.4 percent of all hospital admissions and that diagnostic adverse events accounted for 6.4 percent of all adverse events. The researchers had reviewers classify the causes of diagnostic adverse events by human, organizational, technical, patient-related, and other factors (Zwaan et al., 2010). They further divided the “human” category into knowledge-based, rule-based, skill-based, or other (such as violations or failures by deliberate deviations from rules or procedures). They found that human failures were the main cause of diagnostic adverse events—96.3 percent of these events had a human cause.⁴ However, organizational and patient-related factors were present in 25.0 percent and 30.0 percent of diagnostic adverse events, respectively. The researchers found that the primary causes of diagnostic adverse events were knowledge-based failures (physicians did not have sufficient knowledge or applied their knowledge incorrectly) and information transfer failures (physicians did not receive the most current updates about a patient).

In another study by Zwaan and colleagues (2012), rather than focusing exclusively on adverse events, the researchers had four internists review 247 patient medical records for patients with dyspnea (shortness of breath) symptoms. The reviewers used a questionnaire to identify failures in diagnostic reasoning, diagnostic errors, and harm. They found that failures in diagnostic reasoning occurred in 66 percent of the cases, that diagnostic errors occurred in 13.8 percent of all cases, and that the patient was harmed in 11.3 percent of cases. Although cases with diagnostic errors and patient harm had more failures in diagnostic reasoning, in 4 percent of the cases diagnostic errors occurred in the absence of diagnostic reasoning failures.

Singh et al. (2014) estimated the frequency of diagnostic error in the outpatient setting using data from three prior studies (Murphy et al., 2014; Singh et al., 2010a, 2012a). Two of the studies used “triggered” electronic queries to identify suspected cases of diagnostic error. In one study these triggers identified medical records in which a patient had a primary care visit followed by an unplanned hospitalization or unscheduled follow-up appointment, while the other study looked for a lack of follow-up for abnormal colorectal cancer findings. The third study examined consecutive cases of lung cancer. Physicians reviewed medical records to determine if there was a diagnostic error (defined as a missed opportunity to make or pursue the correct diagnosis when adequate data were available at the index [i.e., first] visit) (Singh, 2012a). The combined estimate of diagnostic error based on these three datasets was about 5 percent. Extrapolating to the entire U.S. population, Singh et al. (2014) estimated that approximately 12 million adults (or 1 in 20 adults) experience a diagnostic error each year; the researchers suggested that about half of these errors could be potentially harmful. Due to the definition of diagnostic error that Singh and colleagues employed, they asserted—as have other researchers—that this number may be a conservative estimate of the rate of outpatient diagnostic errors (Aleccia, 2014).

Opportunities for improvement Medical records will continue to be an important source of data for assessing diagnostic errors. The advent of electronic forms that make some methods more cost-efficient, combined with mechanisms such as health information exchanges that may make it easier to assemble the entire patient diagnostic episode, may enhance the use of these methods. Developing a standard method that could be applied to a random sample of records (either nationally or in prespecified settings) would enhance opportunities to learn about both the incidence and the variation in the likelihood of patients experiencing a diagnostic error. Greater attention to the reliability with which the method is applied, particularly through the use of explicit rather than implicit methods, would also enhance the scientific strength of these studies.

Medical Malpractice Claims

Description of the data source Medical malpractice claims are defined as the electronic and paper databases maintained by professional liability insurers on claims that have been filed by patients or their families seeking compensation for alleged medical errors, including diagnostic errors; the information in support of the claims (medical records, depositions, other reports); and the final determination, whether achieved through a settlement or a court ruling. In addition to files maintained by insurers, the Health Resources and Services Administration, an agency within the Department of Health and Human Services (HHS), maintains the National Practitioner Data Bank (NPDB). The NPDB is a repository of clinician names, affiliations, and malpractice payments that have been made. It serves primarily as a system to facilitate comprehensive review of the credentials of clinicians, health care entities, providers, and suppliers, but it has been used for research as well. Many states also require claim reporting for purposes of maintaining a state-level database of paid claim information.

Notes about the data source For a diagnostic error to be included in malpractice claims datasets, a patient must have filed a claim, which is a relatively rare event (Localio et al., 1991), and is more likely if the patient has experienced significant harm or if negligence is a factor. For example, one study using data from the Harvard Medical Practice Study estimated that the probability of negligent injury was 0.43 percent and that the probability of nonnegligent injury was 0.80 percent (Adams and Garber, 2007). Furthermore, the probability that a claim would be filed was 3.6 percent if a negligent injury occurred and 3.2 percent if a nonnegligent injury occurred. The probability that a claim would be paid was 91 percent for negligent injury claims and 21 percent for nonnegligent injury claims. Thus, malpractice claims data provide a small window into the problem of diagnostic errors and are biased toward more serious diagnostic errors. For diagnosis-related claims, an average of 5 years elapses between the incident and the settlement of the claim (Tehrani et al., 2013). The validity of claims is uncertain; some claims will be filed and closed when no error occurred. Many, if not most, errors do not lead to malpractice claims. Cases may also be dismissed even when a true diagnostic error occurred.

Methods for identifying cases for review (denominator) Studies of diagnostic error using malpractice claims data use all malpractice claims (any allegation) as the denominator.

Methods for determining if an error occurred (numerator) In malpractice claims, the allegation in the claim is the basis for a determination; multiple allegations can be associated with a single claim. A number of studies have assessed the validity of malpractice claims (Localio et al., 1991; Studdert et al., 2000, 2006). Generally speaking, studies use only closed claims, that is, those for which the insurer has determined that no further legal action will be taken (claims may be closed due to settlement, verdict, dismissal, abandonment, or other reasons). Data from CRICO's Comparative Benchmarking System indicate that 63 percent of closed diagnosis-related cases were withdrawn, denied, or dismissed with no indemnity payment (CRICO, 2014).

What is known Tehrani et al. (2013) analyzed 25 years of closed medical malpractice claims from the National Practitioner Data Bank in order to characterize the frequency, patient outcomes, and economic consequences of diagnostic errors. The researchers found that diagnostic errors were the leading type of paid malpractice claims (28.6 percent) and were responsible for the highest proportion of total payments (35.2 percent) (Tehrani et al., 2013). Diagnostic errors were almost twice as likely to be associated with patient death as other allegation categories (such as treatment, surgery, medication, or obstetrics claims). Almost 70 percent of diagnostic error claims were from the outpatient setting, but inpatient diagnostic error claims were more likely to be associated with patient death. The researchers estimated that the 2011 inflation-adjusted mean and median per claim payout for diagnostic error were $386,849 and $213,250, respectively.

Schiff and colleagues (2013) reviewed closed primary care malpractice claims in Massachusetts from 2005 to 2009. During that 5-year period, 551 medical malpractice claims were from primary care practices. More than 70 percent of the allegations were related to diagnosis. The diagnoses most often appearing in these claims were cancer, heart diseases, blood vessel diseases, infections, and stroke.

CRICO has conducted comprehensive analyses of its claim files and associated medical records for diagnostic errors (CRICO, 2014; Siegal, 2014). CRICO's database represents about 30 percent of the NPDB and includes around 400 hospitals and health care entities and 165,000 physicians. In CRICO's analysis of data from 2008 to 2012 (including more than 4,500 cases and more than $1 billion total incurred losses), the organization reported that diagnosis-related claims represented 20 percent of cases by volume and 27 percent of indemnity payments. It found that diagnostic errors are more common in the ambulatory care setting than in the inpatient or emergency department setting (56 percent versus 28 percent and 16 percent, respectively). Within the inpatient setting, the top diagnoses represented in closed malpractice claims included myocardial infarction (MI) and cardiac events, complications of care (failure to rescue), and infections/sepsis (Siegal, 2014). In the ambulatory care setting, cancer, cardiac care (including MI), and injury (orthopedic, head, and spine) represented the top diagnoses in paid claims. CRICO found that cancer represented almost one-third of all the diagnosis-related medical malpractice claims.

The Doctors Company, another large national medical liability insurer, compiled information from its 2007–2013 claims database for the committee. In its analysis of diagnosis-related claims, The Doctors Company included information from 10 medical specialties (internal medicine, family medicine, obstetrics, cardiology, gynecology, general surgery, emergency medicine, orthopedics, pediatrics, and hospital medicine). For the 10 specialties, diagnosis-related claims constituted between 9 percent (obstetrics) and 61 percent (pediatrics) of total claims. The analysis included the top five diagnoses associated with each specialty's malpractice claims. That analysis indicated that more than half of the diagnoses appeared within multiple specialties and generally were for commonly encountered diseases (such as acute MI, acute cerebral vascular accident, cancer, and appendicitis) (Troxel, 2014).

Opportunities for improvement For malpractice claims to be useful for estimating the incidence of diagnostic error, it will be necessary to develop a better understanding of the underlying prevalence of diagnostic error as well as of the probability that a claim will be filed if an error has occurred and the likelihood that a filed claim will be settled. This will require significant research activity, and such research would have to explore variations by geography, specialty, type of error, and other factors. Databases from malpractice insurers contain much more clinical detail than the NPDB and are likely to be more useful in describing patterns of diagnostic errors, such as the steps in the diagnostic process that present the highest risk for different diagnoses. CRICO's benchmarking studies demonstrate the utility of these data for understanding where in the diagnostic process errors are most likely to occur and what factors contributed to the error. This can be useful for designing both monitoring and improvement programs.

Health Insurance Claims

Description of the data source The data source consists of electronic databases maintained by health insurance companies that contain the details of bills submitted by health care professionals and organizations for payment of services delivered. Both public (e.g., Medicare, Medicaid) and private (e.g., Aetna, Blue Cross, United Healthcare) entities maintain such databases on the individuals with whom they have a contractual arrangement to provide payment. Typically, health care professionals and organizations bill multiple insurers for services.

Notes about the data source For information to be present in the database, a patient has to have used a service, a claim must have been filed, the service must have been covered, and (usually) payment must have been made. Claims are based on structured coding systems (ICD-9/10, CPT-IV, NDC, DRG) and do not generally include clinical details (e.g., results of history and physical examinations, diagnostic testing results) except as categorical codes. Because data are available electronically and represent the universe of claims filed for any insurer, the probability that a patient or episode of care has been selected for analysis can be calculated. Because health care professionals and organizations bill multiple insurance companies, each of which has different rules, it can be difficult to understand the health care professionals' and organizations' overall practices with data from a single source.

Methods for identifying cases for review (denominator) Although a random sample of claims or groups of claims could be selected, it is more common to focus studies on those with patterns of care consistent with the possibility that a diagnostic error occurred.

Methods for determining if an error occurred (numerator) Frequently, an algorithm is developed to determine when an error likely occurred, such as cases in which there is no evidence that a diagnostic test was done prior to a new diagnosis being made (e.g., breast cancer diagnosis in the absence of a screening mammogram). Health insurance claims data may be linked to other data sources (e.g., National Death Index, diagnostic testing results, medical records) to make a determination that an error occurred.

What is known Within the quality and safety field, improvements in the measurement of both process and outcome measures of quality have been made possible by the expanding use of health information technology (health IT) and health insurance claims databases over the past several decades. For example, health insurance claims databases linked to validated federal death registries have made possible the measurement of 30-day mortality for acute MI, heart failure, and pneumonia, all of which are considered as outcome measures of quality. Similar databases provide the backbone for measuring process quality measures (such as 30-day rehospitalizations, appropriate assessment of left ventricular function in patients with congestive heart failure, and retinopathy screening among patients with diabetes). There are a few examples of the use of these data for investigating diagnostic error. Newman-Toker and colleagues (2014) identified patients who were admitted to the hospital with a diagnosis of stroke who in the previous 30 days had been treated and released from an emergency department for symptoms consistent with a stroke. They found that 12.7 percent of stroke admissions reflected potential missed stroke diagnoses and 1.2 percent reflected probable missed diagnoses. These rates suggest that 15,000 to 165,000 stroke diagnoses are missed annually in the United States, with a higher risk for missed diagnoses among younger, female, and white patients. The researchers note that their estimates of diagnostic error are inferred rather than confirmed because of the lack of clinical detail in health insurance claims.

Opportunities for improvement Health insurance claims databases maintained by the Centers for Medicare & Medicaid Services (CMS) and by commercial insurers offer the possibility of measuring certain types of diagnostic errors, identifying their downstream clinical consequences and costs, and understanding the system-level, health care professional–level, and patient-level factors that are associated with these errors.

For example, analyses of claims data could be used in “look back” studies to identify the frequency with which acute coronary syndrome is misdiagnosed. Specifically, for those enrollees who are ultimately diagnosed with acute coronary syndrome, analysts could explore how frequently these beneficiaries were seen by health care professionals in the week prior to ultimate diagnosis (either in outpatient, emergency department, or hospital settings), the incorrect diagnoses that were made, and the factors associated with the diagnostic error. For instance, this epidemiologic approach using large administrative databases would make it possible to determine whether the diagnostic error occurs more frequently in specific hospitals, among specific types of clinicians or practice settings, or during particular days of the week when staffing is low or the volume of patients treated is unexpectedly high. The strength of this approach to understanding the epidemiology of diagnostic error is its ability to provide national estimates of diagnostic error rates across a vast array of conditions; to understand how these diagnostic error rates vary across geography and specific settings of care; to study the impact of specific care delivery models on diagnostic error rates (e.g., do accountable care organizations lower diagnostic errors?); and to update measurements as quickly as the administrative data are themselves collected. The main critique of this approach concerns the validity of the findings because of the limited availability of the clinical data necessary to confirm a diagnosis. Thus, this data source may be most useful in combination with other sources.

Diagnostic Testing (Anatomic and Clinical Pathology)

Description of the data source Diagnostic testing includes the examination of secretions, discharges, blood, or tissue using chemical, microscopic, immunologic, or pathologic methods for the purposes of making or ruling out a diagnosis. Analysis of the data may involve automated processes or a visual examination by trained health care professionals (clinical and anatomic pathologists).

Notes about the data source A unique feature of this type of data is that the original source data (the samples) are frequently available for reanalysis or inspection by another health care professional, thus allowing for an independent assessment based on the same data. For the committee's purposes, the focus is on those diagnoses for which diagnostic testing findings are a key information source. A common taxonomy in this field distinguishes among five phases: pre-pre-analytic (i.e., deciding whether or not to order a particular test), pre-analytic (i.e., sample labeling and acquisition, test performance), analytic (i.e., the accuracy of the test or examination of the sample), post-analytic (i.e., the results are reported correctly, interpreted correctly, and communicated back to the ordering clinician in a timely way), and post-post-analytic (i.e., the ordering clinician uses test results to inform patient care) (Plebani et al., 2011). For the purpose of examining the incidence of diagnostic error, the committee focused on those circumstances in which diagnostic testing results are a key information source. One study estimated that at least 10 percent of diagnoses require diagnostic testing results in order to be considered final; this number is likely higher today (Epner et al., 2013; Hallworth, 2011; Peterson et al., 1992). Primary care clinicians order tests in about one-third of patient visits (Hickner et al., 2014). For anatomic pathology specimens, which require visual inspection and clinical judgment, second reviews by another pathologist offer insight into the potential rate of diagnostic error.

Methods for identifying cases for review (denominator) Two methods—random samples and prespecified criteria—are commonly used to identify cases. Both methods allow for the denominator to be characterized (i.e., the probability that a case was reviewed, the characteristics of the cases reviewed as compared to all cases).

Methods for determining if an error occurred (numerator) Because testing involves multiple steps, there are many different methods for identifying errors, including an examination of other data sources such as medical records, malpractice claims, or pharmacy databases (Callen et al., 2011). For second review studies, an error is typically defined as a discrepancy between the findings of the first pathologist and the second pathologist. This review can identify errors in which a finding that leads to a diagnosis was missed and errors in which a finding was inaccurate (i.e., no disease was found by the second reviewer). Second review studies typically assume that the second review is more accurate, but these studies do not typically link to patient outcomes. When second reviews are linked to patient outcomes, Renshaw and Gould (2005) concluded that in many cases, the first reviewer was correct. For other diagnostic tests, errors may be detected in the interpretation or communication of results in a timely manner.

What is known Plebani reported that errors in laboratory medicine studies vary greatly because of the heterogeneity in study designs and the particular step or steps in the process that were examined (Plebani, 2010). A considerable focus on the analytic phase has led to substantial reductions in errors in that step; the pre- and post-analytic phases are seen as more vulnerable to error. A review published in 2002 (that only classified the diagnostic testing process in three phases) found that 32 to 75 percent of errors occurred in the pre-analytic phase, 13 to 32 percent in the analytic phase, and 9 to 31 percent in the post-analytic phase (Bonini et al., 2002). A study of urgent diagnostic testing orders in the hospital, which also classified the diagnostic testing process in three phases, found that 62 percent of errors were in the pre-analytic phase, 15 percent in the analytic phase, and 23 percent in the post-analytic phase (Carraro and Plebani, 2007). One study estimated that 8 percent of errors had the potential to result in serious patient harm (Goldschmidt and Lent, 1995). A systematic review of the literature on follow-up of test results in the hospital found failure rates of 1 to 23 percent in inpatients and 0 to 16.5 percent in emergency department patients (Callen et al., 2011).

As Berner and Graber (2008) note, second reviews in anatomic pathology identify varying discrepancy rates. The College of American Pathologists and the Association of Directors of Anatomic and Surgical Pathology recently published guidelines based on a systematic review of the literature which found a median rate of major discrepancies in 5.9 percent of cases (95 percent confidence interval, 2.1–10.5 percent) (Nakhleh et al., 2015). The study also reported variations in the rate by the service performed (surgical pathology versus cytology), the organ system (single versus multiple), and the type of review (internal versus external). Kronz and Westra (2005) report a diagnostic discrepancy rate for the head and neck found by second review of between 1 and 53 percent for surgical pathology and 17 to 60 percent for cytopathology. A study by Gaudi and colleagues (2013) found that pathologists with dermatopathology fellowship training were more likely to disagree with preliminary diagnoses provided by nonspecialist pathologists.

Opportunities for improvement The contribution of diagnostic testing to diagnosis is substantial, but it has not been systematically quantified recently. The understanding of this critical information source could be improved by developing better methods for identifying and enumerating the diagnoses for which such testing is critical, mechanisms for evaluating the appropriateness of test ordering, and methods for determining the impact on patient outcomes. Additionally, studies that use diagnostic variance as a surrogate for accuracy (second reviews in which the second reviewer is considered more accurate) could benefit from the inclusion of patient outcomes.

Medical Imaging

Description of the data source The data are visual representations of the interior of the body generated using a variety of methods (e.g., X-ray, ultrasound, computed tomography [CT], magnetic resonance imaging, and positron emission tomography) that are collected for the purpose of diagnosis; these visual representations generally require interpretation by a radiologist or, in certain circumstances, physicians in nuclear medicine, emergency medicine, or cardiology. In this context, the medical imaging data are reviewed by at least one other clinician, and the findings of all health care professionals are recorded.

Notes about the data source As with anatomic pathology, a unique feature of this data type is the availability of the original images for review by a second radiologist. The focus is on those diagnoses for which medical imaging results are a key information source. In approximately 15 percent of office visits, an imaging study is ordered or provided (CDC, 2010), whereas one or more medical imaging studies are ordered in approximately 47 percent of emergency department visits (CDC, 2011). In both settings, X-rays are the most common imaging method used.

Methods for identifying cases for review (denominator) Typically a random sample of cases is selected for second review, although some studies have included prespecified criteria (e.g., cases known to have higher potential rates of error in interpretation, or abnormal findings only).

Methods for determining if an error occurred (numerator) An error is assumed to have occurred whenever a discrepancy exists between the two clinicians in interpreting the medical imaging study. Some studies have also involved radiologists conducting a second review of their own previously completed studies.

What is known Berlin noted that medical imaging discrepancy rates as indicated by second review have not changed much over the past 60 years (Berlin, 2014). For instance, a study by Abujudeh and colleagues explored intra- and interobserver variability in medical imaging by having three experienced radiologists review 30 of their own previously interpreted CT exams and 30 CT exams originally interpreted by other radiologists (Abujudeh et al., 2010). They found a major discrepancy rate of 26 percent for interobserver variability and 32 percent for intraobserver variability. Velmahos and colleagues (2001) found an 11 percent discrepancy rate between the preliminary and final readings of CT scans of trauma patients. Discrepancy rates were negatively associated with level of experience: The lower the level of experience of the preliminary reader, the more likely there was to be a discrepancy. In many of the second review studies in imaging, high error rates resulted from using a denominator that consisted only of abnormal cases. Studies that look at real-time errors—that is, devising an error rate using both normal and abnormal exams as the denominator—suggest an error rate in the 3 to 4.4 percent range (Borgstede et al., 2004).

Opportunities for improvement Medical imaging plays a key role in many diagnoses, and errors in the use and interpretation of these studies can contribute to diagnostic error. For the purposes of estimating the incidence of diagnostic error due to errors related to medical imaging, it would be useful to identify the subset of diagnoses for which medical imaging results are central to making the diagnosis and to conduct studies to determine the likelihood of errors, the nature of those errors, and the variation in the circumstances under which errors occur. The role of second reviews in error recovery—identifying and “intercepting” errors before they affect patient outcomes—both for medical imaging and for anatomic pathology is discussed in Chapter 6.

Surveys of Clinicians

Description of the data source The data come from questionnaires (written, telephone, interview, Web-based) that obtain clinicians' self-reports about diagnostic errors they have made or what they know about diagnostic errors made by other clinicians. The information content of such surveys can vary.

Notes about the data source As with all surveys, the results can be affected by a number of biases, including nonresponse bias (nonresponders being systematically different from responders, such as being more or less likely to have committed a diagnostic error) or reporting bias (systematic differences in the information that is revealed or suppressed, such as not reporting more serious errors). Unless the self-report can be compared to an authoritative source, it is difficult to determine the validity of rates based solely on self-report. Surveys usually have the advantage of anonymity, which might make respondents more likely to report their errors accurately than through other methods.

Methods for identifying cases for review (denominator) Surveys are frequently conducted on random samples of clinicians, making the implicit denominator the number of opportunities a clinician had to make a diagnosis in the study period. Convenience samples are also used (e.g., surveys of clinicians participating in a continuing medical education course). Reports of survey findings have used different denominators, but often the denominator is the number of clinicians responding to the survey.

Methods for determining if an error occurred (numerator) An error is judged to have occurred when a clinician self-reports having made one or more diagnostic errors in the study time frame. Some studies have asked about errors known to the clinician that were made by other clinicians or experienced by family members. This approach makes estimating the incidence rate nearly impossible, as the true denominator is unknown.

What is known Schiff et al. (2009) surveyed physicians and asked them to recall instances of diagnostic error. In their analysis of 583 reports of diagnostic error, they found that physicians readily recalled instances of diagnostic error; the most commonly reported diagnostic errors were pulmonary embolism, drug reactions, cancer, acute coronary syndrome, and stroke. Singh and colleagues (2010b, p. 70) surveyed pediatricians about diagnostic errors and found that “more than half of respondents reported that they made a diagnostic error at least once or twice per month.” In another survey of physicians, 35 percent reported that they had experienced medical errors either in their own or a family member's care (Blendon et al., 2002).

Opportunities for improvement For the purposes of making national estimates of the incidence of diagnostic errors, it would be useful to have more clearly defined sampling frames, more detailed questions about the nature of the errors and the circumstances surrounding the error, and an opportunity to compare this method to other methods that use different data sources. Surveys have the advantage of being a potentially easy way to get a snapshot of diagnostic error rates, but the quality of the information may make this source less useful for other applications. The biases that are inherent in surveys are difficult to overcome and likely limit the utility of this source.

Surveys of Patients

Description of the data source The data come from questionnaires (written, telephone, interview, Web-based) that obtain patients' self-reports about diagnostic errors they have experienced or their awareness of diagnostic errors experienced by others. The information collected can vary.

Notes about the data source As with all surveys, the results can be affected by nonresponse bias and by reporting bias. Unless there are opportunities to compare answers to other data sources, it may not be possible to confirm the validity of the responses. Patient definitions of diagnostic errors might vary from the definitions of health care professionals. Patient surveys can be very useful in determining whether a new health problem was explained to the patients and whether they understood the explanation.

Methods for identifying cases for review (denominator) Surveys are usually conducted on a sample of patients that is randomly drawn from some population (e.g., geographic area, members of a health plan, and patients who utilize a specific care setting) or selected so that the patients meet certain criteria (similar to the trigger tools discussed above). Convenience samples are also used.

Methods for determining if an error occurred (numerator) The determination of an error is based on self-report by the patient. Some studies inquire about both the patient's own experience and that of others known to the patient. The latter approach makes it impossible to estimate a true incidence rate because of uncertainty around the real size of the denominator.

What is known In one survey of patients, 42 percent reported that they had experienced medical errors either in their own or a family member's care (Blendon et al., 2002). A poll commissioned by the National Patient Safety Foundation found that approximately one in six of those surveyed had experience with diagnostic error, either personally or through a close friend or relative (Golodner, 1997). More recently, 23 percent of people surveyed in Massachusetts indicated that they or someone close to them had experienced a medical error, and approximately half of these errors were diagnostic errors (Betsy Lehman Center for Patient Safety and Medical Error Reduction, 2014). Weissman and colleagues (2008) surveyed patients about adverse events during a hospital stay and compared survey-detected adverse events with medical record review. Twenty-three percent of surveyed patients reported at least one adverse event, compared to 11 percent identified by medical record review.

Opportunities for improvement The particular value of patient surveys is likely to be related to understanding failures at the front end of the diagnostic process (failure to engage) and in the process of delivering an explanation to the patient. Both are critical steps, and patients are uniquely positioned to report on those elements of diagnostic performance. The committee did not have examples of this application, and potential future uses are discussed in Chapter 8.

Other Methods

A variety of other methods have been employed to examine different dimensions of diagnostic error. These methods were not included in the table because they are unlikely to be a major source for estimating the incidence of error.

Patient actors, or “standardized patients,” have been used to assess rates of diagnostic error. Patient actors are asked to portray typical presentations of disease, and clinicians are assessed on their diagnostic performance. In one study in internal medicine, physicians made diagnostic errors in 13 percent of interactions with patient actors portraying four common conditions (Peabody et al., 2004). In a more recent multicenter study with unannounced patient actors, Weiner et al. (2010) looked at both biomedical-related errors (such as errors in diagnosis and treatment) and context-related errors (such as the lack of recognition that a patient may be unable to afford a medicine based on certain patient cues) in patient management. They found that physicians provided care that was free from errors in 73 percent of the uncomplicated encounters but made more errors in more complex cases (Weiner et al., 2010).

Many health care organizations in the United States have systems in place for patients and health care professionals to report minor and major adverse events. However, voluntary reporting typically results in under-reporting and covers only a limited spectrum of adverse events (AHRQ, 2014b). For example, one study found that over half of voluntary reports concentrated on medication/infusion adverse events (33 percent), falls (13 percent), and administrative events, such as discharge process, documentation, and communication (13 percent) (Milch et al., 2006). In Maine, the use of a physician champion to encourage voluntary diagnostic error reporting was implemented in 2011. During the 6-month pilot, there were 36 diagnostic errors reported. Half of the diagnostic errors were associated with moderate harm, and 22 percent of the diagnostic errors were classified as causing severe harm (Trowbridge, 2014).

Direct observation is another method that has been used to identify medical errors. Andrews and colleagues (1997) conducted observational research within a hospital setting and found that approximately 18 percent of patients in the study experienced a serious adverse event.

There have also been efforts to assess disease-specific diagnostic error rates, using a variety of data sources and methods. Berner and Graber (2008) and Schiff and colleagues (2005) provide examples of diagnostic errors in a variety of disease conditions.

Summary of Approaches to Assess the Incidence of Diagnostic Error

A number of methods have been used to assess the frequency with which diagnostic error occurs. Based on the committee's review, the most promising methods for estimating incidence are postmortem exams, medical record reviews, and medical malpractice claims analysis, but none of these alone will give a valid estimate of the incidence of diagnostic error. This conclusion is consistent with studies in the broader area of medical errors and adverse events. For example, the Office of Inspector General of HHS completed an analysis that compared different measurement methods (nurse reviews, analysis of administrative claims data, patient interviews, analysis of incident reports, and an analysis of patient safety indicators) and found that 46 percent of patient safety events were identified by only one of the methods (Office of Inspector General, 2010). Levtzion-Korach and colleagues (2010) compared information gathered with five different measurement approaches—incident reporting, patient complaints, risk management, medical malpractice claims, and executive WalkRounds—and concluded that each measurement method identified different but complementary patient safety issues. In a related commentary, Shojania concluded that “it appears that a hospital's picture of patient safety will depend on the method used to generate it” (Shojania, 2010, p. 400). This suggests that no one method will perfectly capture the incidence and the nature of medical errors and adverse events in health care: “[A] compelling theme emerged . . . different methods for detecting patient safety problems overlap very little in the safety problems they detect. These methods complement each other and should be used in combination to provide a comprehensive safety picture of the health care organization” (Shekelle et al., 2013, p. 416). This likely applies to the measurement of diagnostic errors; with the complexity of the diagnostic process, multiple approaches will be necessary to provide a more thorough understanding of the occurrence of these errors.

The Diagnostic Process and Measurement Approaches to Identifying Potential Failures

Because the diagnostic process is a complex, team-based, iterative process that occurs over varying time spans, there are numerous opportunities for failures. The failures can include (1) the step never occurring, (2) the step being done incompletely or incorrectly (accuracy), and (3) a meaningful delay in taking a step (timeliness). In Figure 3-2, the committee's conceptual model is used to identify where in the diagnostic process these failures can occur, including the failure of engagement in the health care system, failure in the diagnostic process, failure to establish an explanation of the health problem, and failure to communicate the explanation of the health problem.

FIGURE 3-2

Places in the diagnostic process where failures can occur that contribute to diagnostic errors.

Table 3-2 is organized around the major steps in the diagnostic process and adapts Schiff and colleagues' (2009) framework to the failures associated with each of these steps. For example, diagnostic testing is part of several diagnostic steps where failures may happen, namely, during information gathering, integration, and interpretation. The last column identifies some of the methods that can be used to identify failures in actual practice settings. Experimental laboratory methods are a complementary approach to the methods in Table 3-2 to understand potential failures related to reasoning (Kostopoulou et al., 2009, 2012; Zwaan et al., 2013). The following discussion includes more information about the measurement approaches that can be used at each of these steps.

TABLE 3-2

Methods for Detecting Failures Across the Diagnostic Process.

Failure of engagement This step primarily involves either patients not recognizing symptoms or health risks rapidly enough to access the health care system or patients experiencing significant barriers to accessing health care. Health care organizations are familiar with routine measures of eligible patients presenting for common screening tests; these systems can be extended to detect other failures to engage (or reengage) related to routine monitoring for disease progress, follow-up of abnormal test results, and so on (Danforth et al., 2014; Kanter, 2014; Singh et al., 2009). Surveys and interviews with patients can be used to identify approaches that are likely to be successful (and unsuccessful) in reducing delays and increasing engagement. The CRICO benchmarking study found that 1 percent of malpractice claims had an error associated with a failure to engage (CRICO, 2014).

Failure in information gathering The information-gathering step can involve failures to elicit key pieces of information; a failure to order the right diagnostic testing (in the right sequence or with the right specification); or technical errors in the way that samples are handled, labeled, and processed. The CRICO benchmarking study found that 58 percent of cases had one or more errors in the initial diagnostic assessment (CRICO, 2014). Failure to order appropriate diagnostic tests has been found to account for 55 percent of missed or delayed diagnoses in malpractice claims in ambulatory care (Gandhi et al., 2006) and 58 percent of errors in emergency departments (Kachalia et al., 2006). In their examination of physician-reported cases of error, Schiff and colleagues (2009) found that a failure or delay in ordering needed tests was the second most common factor contributing to a diagnostic error. Methods of rapid detection might include random reviews, diagnostic trigger tools, checklists, observation, video or audio recording, and feedback.

Failure in interpretation Inaccurate or failed attempts to interpret information gathered in the diagnostic process can involve such things as diagnostic tests, clinical history and interview, or information received from referral and consultation with other clinicians. CRICO reported that 23 percent of cases in its malpractice benchmarking study had errors in diagnostic test interpretation; 49 percent had errors in medical imaging, 20 percent in medicine, 17 percent in pathology, and 8 percent in surgery (CRICO, 2014). Schiff and colleagues (2009) reported that an erroneous laboratory or radiology reading of a test contributed to 11 percent of the diagnostic errors that they examined. Studies have shown that an incorrect interpretation of diagnostic tests occurs in internal medicine (38 percent reported in Gandhi et al., 2006) and emergency medicine (37 percent reported in Kachalia et al., 2006). Hickner and colleagues (2008) found that 8.3 percent of surveyed primary care physicians reported uncertainty in interpreting diagnostic testing. Failure in interpretations for medical imaging and anatomic pathology can be identified through second reviews conducted by expert clinicians.

Failure in integration Integration failures can be divided into failures in hypothesis generation, the suboptimal weighting and prioritization of information gathered in the diagnostic process, and the failure to recognize or weight urgency of clinical signs or symptoms. In examining major diagnostic errors, Schiff and colleagues (2009) found that 24 percent were the result of a failure to consider or a delay in considering the correct diagnosis. Potential approaches to measuring failure in integration include structured debriefings with the clinicians involved, conferences that review diagnostic errors (such as morbidity and mortality [M&M] conferences and root cause analyses), and random reviews.

Failure to establish an explanation (diagnosis) Failures can also occur when there is a failure to establish the explanation of the patient's health problem. This can include suboptimal weighting and prioritization of clinical signs and symptoms, delays in considering a diagnosis, or failing to follow up with patients (including failing to create and implement an appropriate follow-up plan). CRICO (2014) found that referral errors were common in cancer cases in which there were diagnostic errors (48 percent of cases lacked appropriate referrals or consults). Methods for identifying these failures include random reviews and the analysis of expected follow-up, such as Kaiser Permanente's SureNet system (Danforth et al., 2014; Graber et al., 2014).

Failure to communicate the explanation Failures to communicate the explanation of a patient's health problem can include cases in which no communication was attempted, in which there was a delay in communicating the explanation, or in which the communication occurred but it was not aligned with a patient's health literacy and language needs and was not understood. CRICO (2014) reported that 46 percent of cases in its benchmarking study involved a failure in communication and followup, including 18 percent of cases where the clinician did not follow up with the patient and 12 percent of cases where the information was not communicated within the care team. Potential measurement methods for this step include video recording and debriefing, patient surveys, medical record reviews, and shared decision-making results.

Other researchers have employed different classification schemes to illustrate where in the diagnostic process failures occur. For example, some researchers have classified the diagnostic process into three phases: initial diagnostic assessment; diagnostic test performance, interpretation, and results reporting; and diagnostic follow-up and coordination (CRICO, 2014; Lyratzopoulos et al., 2015). Another framework that is useful to depict the steps in the diagnostic testing process where failures can occur is the brain-to-brain loop model described in Chapter 2. The nine-step process was originally developed in the laboratory medicine setting (Lundberg, 1981; Plebani et al., 2011), but it can be applied to anatomic pathology and medical imaging as well. Targeted measurement has shown that the phases of the process that are most prone to errors occur outside of the analytical phase and include test ordering (part of the diagnostic process information-gathering step) and subsequent decision making on the basis of the test results (part of the interpretation step) (Epner et al., 2013; Hickner et al., 2014; Plebani et al., 2011).

The Work System and Measurement Approaches to Identifying Potential Vulnerabilities and Risk Factors

In considering the options for making significant progress on the problem of diagnostic error, it is important to understand the reasons why these failures occur. For this discussion, the committee draws on the general patient safety literature, and applies it specifically to the challenge of diagnostic error. Traditional approaches to evaluating medical errors have focused on identifying individuals at fault. However, the modern patient safety movement has emphasized the importance of a systems approach to understanding medical errors. According to the IOM report To Err Is Human: Building a Safer Health System:

The common initial reaction when an error occurs is to find and blame someone. However, even apparently single events or errors are due most often to the convergence of multiple contributing factors. Blaming an individual does not change these factors and the same error is likely to recur. Preventing errors and improving patient safety for patients require a systems approach in order to modify the conditions that contribute to errors. People working in health care are among the most educated and dedicated workforce in any industry. The problem is not bad people; the problem is that the system needs to be made safer. (IOM, 2000, p. 49)

Often, a diagnostic error has multiple contributing factors. One analogy that has been employed to describe this phenomenon is the Swiss cheese model developed by psychologist James Reason (AHRQ, 2015a; Reason, 1990). In this model, a component of the diagnostic process would represent a slice of cheese in a stack of slices. Each component within the diagnostic process has vulnerabilities to failure (represented by the holes in a slice of Swiss cheese); in a single step of the diagnostic process, this may not affect the outcome. However, if the vulnerabilities (holes in the Swiss cheese) align, a diagnostic error can result.

Another way to think about the causes of diagnostic error is to distinguish between active errors and latent errors. Active errors typically involve frontline clinicians (sometimes referred to as the “sharp end” of patient safety) (IOM, 2000). In contrast, latent errors are more removed from the control of frontline clinicians and can include failures in organizations and design that enable active errors to cause harm (often called the “blunt end” of patient safety) (AHRQ, 2015a; IOM, 2000). In the event of a medical error, too often the focus is on identifying active errors, especially within health care organizations with punitive cultures that focus on individual blame and punishment. But the IOM noted that:

Latent errors pose the greatest threat to safety in a complex system because they are often unrecognized and have the capacity to result in multiple types of active errors. . . . Latent errors can be difficult for people working in the system to notice since the errors may be hidden in the design of routine processes in computer programs or in the structure or management of an organization. People also become accustomed to design defects and learn to work around them, so they are often not recognized. (IOM, 2000, p. 55)

In line with the IOM's earlier work, the committee took a systems approach to understanding the causes and risks of diagnostic errors. Consistent with the committee's conceptual model, measurement for this purpose examines the different dimensions of the work system to identify the circumstances under which diagnostic errors are more (and less) likely to occur and to identify the risk factors for such errors. Factors contributing to diagnostic errors can be mapped along the components of the work system, including diagnostic team members and their tasks, technologies and tools, organizational characteristics, the physical environment, and the external environment.

Some of the more familiar approaches for assessing the system causes of medical errors are M&M conferences that apply a modern patient safety framework (a focus on understanding contributing factors rather than a focus on individual errors and blame) (Shojania, 2010) and root cause analyses (AHRQ, 2015b). For example, root cause analysis methods were applied to identify the factors that contributed to delays in diagnosis in the Department of Veterans Affairs system (Giardina et al., 2013). Diagnostic errors have also been evaluated in M&M conferences (Cifra et al., 2015).

As the committee's conceptual model shows, the diagnostic process is embedded in a work system. Examining how the various dimensions of the work system contribute to diagnostic errors or how they can be configured to enhance diagnostic performance leads to a deeper understanding of the complexity of the process. Table 3-3 identifies the dimensions of the work system, the contribution each makes to diagnostic errors, and examples of measurement methods that have been used to assess each area. Although diagnostic team members are a critical component of the work system, approaches to ensuring diagnostic competency are addressed in Chapter 4 and they are not included here. The focus here is on the specific measurement tools that are available to help health care organizations better identify aspects of the work system that present vulnerabilities for diagnostic errors. A distinctive feature of some of these methods is that they can be used proactively to identify risks before an error occurs, versus the measurement methods described above that examine steps leading to an error that has already occurred.

TABLE 3-3

Methods for Assessing the Effect of the Work System on Diagnostic Errors.

Tasks and workflow The diagnostic process involves a series of tasks and an implicit or explicit workflow that contains and connects those tasks. A variety of challenges can occur with the tasks and workflow that are required to make a diagnosis, including problems with the information (amount, accuracy, completeness, appropriateness), communication issues, the complexity of the task, a lack of situational awareness, poor workflow design, interruptions, and inefficiencies. These issues contribute to diagnostic error at each step in the information gathering, integration, and interpretation process; they can contribute to problems with the timeliness of information availability, and they can lead to problems in cognitive processing.

There are a variety of measurement approaches that can be used to evaluate tasks and workflow. It should be noted that these are best applied in the real-world environment in which the diagnosis is being made. The methods include cognitive task and work analysis (Bisantz and Roth, 2007; Rogers et al., 2012; Roth, 2008); observation of care processes (Carayon et al., 2014); situation awareness (Carayon et al., 2014; Salas et al., 1995); workflow modeling (Kirwan and Ainsworth, 1992); and proactive risk assessment (Carayon et al., 2014). These methods are briefly described below.

Cognitive task and work analysis The purpose of cognitive task and work analysis is to identify and describe the cognitive skills that are required to perform a particular task, such as making a diagnosis. The most common method used for such an analysis is an in-depth interview combined with observations of the specific task of interest (Schraagen et al., 2000). Because cognitive errors are an important contributing factor to diagnostic errors (Croskerry, 2003) these methods are likely to have considerable utility in efforts to reduce errors. Koopman and colleagues (2015) used cognitive task analysis to examine the relationship between the information needs that clinicians had in preparing for an office visit and the information presented in the electronic health record. They found a significant disconnect between clinician needs and the amount of information and the manner in which it was presented. This disconnect can lead to cognitive overload, a known contributor to error (Patel et al., 2008; Singh et al., 2013). The researchers recommended significant reengineering of the clinical progress note so that it matched the workflow and information needs of primary care clinicians.

Observation of care processes Process observation is a means of verifying what exactly occurs during a particular process (CAHPS, 2012). Frequently, these observations are documented in the form of process maps, which are graphical representations of the various steps required to accomplish a task. The approach is able to capture the complex demands imposed on members of the diagnostic team, and it allows for the “documentation of the coordination and communication required between clinicians to complete a task, use their expertise, tools, information and cues to problem solve” (Rogers et al., 2012). For example, Fairbanks and colleagues (2010) used this method to examine workflow and information flow in an emergency department's use of digital imaging by applying both hierarchical task analysis and information process diagrams. The analysis identified gaps in how the information system for imaging supported communication between radiologists and emergency department physicians. In analyzing diagnostic error, this technique can identify the role that contextual or social factors play in assisting or impeding problem resolution (Rogers et al., 2012). Observations of care processes can also provide input for other work system analysis methods, such as cognitive task and work analysis as well as failure mode and effects analysis (FMEA).

Situation awareness Endsley (1995, p. 36) defined situation awareness as “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.” Situation awareness has been applied at the individual, team, and system levels. There are a variety of approaches to measuring situation awareness, including objective and subjective measures, performance and behavioral measures, and process indices. Because of the multidimensional nature of the construct, a combination of approaches is likely most useful. Examples of measurement tools in medicine include the Anesthetists' Non-Technical Skills (ANTS) measure (Fletcher et al., 2003), the Ottawa Global Rating Scales (Kim et al., 2006), and an instrument to measure pediatric residents' self-efficacy skills (which include situation awareness) in crisis resource management (Plant et al., 2011).

Workflow modeling Workflow modeling is a form of prospective analysis used to describe the processes and activities involved in completing clinical tasks. In contrast to observing work processes, modeling techniques allow for quantitative and qualitative estimations of tasks and of the possible paths that can be taken to complete them (Unertl et al., 2009). Challenges to workflow modeling in health care—and diagnosis in particular—include the fact that clinicians must remain flexible because of the need to respond to the nonroutine presentation of symptoms, results, and events as well as the variability in workflow across different health care organizations. Resulting models can be adapted and modified as necessary to reflect observations of care processes. Numerous methods for workflow modeling exist. Carayon et al. (2012) describe 100 methods in 12 categories (e.g., data display/organization methods and process mapping tools) for workflow modeling of the implementation of health IT. Jun et al. (2009) focus on eight workflow or process modeling methods that have been used in quality improvement projects; these include flowcharts and communication diagrams. These methods have great potential for helping to understand the dynamic sequences of tasks performed by various team members in the diagnostic process.

Proactive risk assessment The term “proactive risk assessment” refers to a variety of methods that are used to identify, evaluate, and minimize potential risks or vulnerabilities in a system. An example of such a method is FMEA. Several steps are involved in FMEA, including graphically describing the process, observing the process to ensure that the diagram is an accurate representation, brainstorming about failure modes, conducting a hazard analysis (i.e., different ways in which a particular process can fail to achieve its purpose), and development of a plan to address each failure mode along with outcome measures. DeRosier and colleagues (2002) describe the use of this method by the Department of Veterans Affairs (VA) National Center for Patient Safety and provide concrete examples of its application.

Technology A variety of technologies are used in the diagnostic process, and these can contribute to diagnostic errors for a variety of reasons, including inappropriate technology selection, poor design, poor implementation, use error, technology breakdown or failure, and misuse of automation. Technology failures contribute to problems in information gathering, integration, and interpretation; they may also produce information overload and may interfere with cognitive processes because of problems with the way the information is received and displayed.

Methods for improving the selection, design, implementation, and use of technology involve some of the methods described above, such as workflow modeling, FMEA, and other proactive risk assessment methods. In particular, many health care organizations have been concerned about whether enough attention is being paid to the usability of health IT. For example, in a study of physician job satisfaction, Friedberg and colleagues (2013) found that a number of factors related to electronic health records (EHRs) had a substantial impact on satisfaction, including: poor usability, the time required for data entry, interference in patient interactions, greater inefficiencies in workflow, less fulfilling work content, problems in exchanging information, and a degradation of clinical documentation. This study used a mixed-method design which included semi-structured and structured interviews with physicians. Its findings were consistent with research using other methods to assess the extent to which EHRs are enhancing care delivery (Armijo et al., 2009; Unertl et al., 2009). The American Medical Informatics Association Board of Directors issued recommendations about improving the usability of EHRs that were based in large part on usability studies that had been conducted by Middleton and colleagues (2013). The use of various usability evaluation methods can help in ensuring that usability concerns are addressed as early as possible in the design process. For example, Smith and colleagues incorporated usability testing into the design of a decision-support software tool to catch missed follow-up of abnormal cancer test results in the VA (Smith et al., 2013). These various possible usability evaluation methods include heuristic evaluation methods, scenario-based usability evaluation, user testing, and the observation of technology in use (Gosbee and Gosbee, 2012).

Organizational characteristics Culture, leadership, and management are some of the organizational characteristics that can affect the diagnostic process. Some of the culture-related issues that can contribute to diagnostic error are a lack of organizational support for improvements, conflicting messages about regulations, confusion about task responsibilities, and the perception by people that they should not speak up even when they know a problem is occurring. These issues have been identified in the broader context of patient safety but are likely to affect diagnostic processes as well.

The main mechanisms for assessing these organizational characteristics are surveys (about culture, leadership, management, collaboration, communication) and focus groups. For instance, Shekelle and colleagues (2013) identified a number of survey-based measures in these areas as part of a report on the context-sensitivity of patient safety practices.

Physical environment Various characteristics of the physical environment (e.g., noise, lighting, layout) may affect the diagnostic process (Alvarado, 2012; Parsons, 2000). The physical environment places additional stresses on a diagnostic team that can affect the performance of cognitive tasks and information gathering, integration, and interpretation. For example, the layout and lighting of the radiology reading room may hinder accurate viewing of screens. Emergency departments are another example of a place where it makes sense to examine the effects of the physical environment on diagnostic errors (Campbell et al., 2007).

Human factors/ergonomics methods can be used to evaluate the physical environment. These methods include, for example, making a direct assessment of noise and lighting with specific equipment (e.g., a light meter) and direct observation of care processes to identify challenges related to layout. For instance, observing the physical movements of clinicians can help identify communication among team members and the barriers posed by the physical environment (e.g., lack of available equipment or poorly located equipment; see Potter et al., 2004; Wolf et al., 2006). In addition, surveys can also be used to gather data from a larger population of staff and patients about environmental characteristics, such as the adequacy of lighting and the perception of noise and its impact. In an example of this approach, Mahmood and colleagues (2011) surveyed nurses about the aspects of their physical environment that affected the risk of medication errors. Many of these factors contribute to latent errors—for example, creating conditions under which cognitive functioning is impaired because of the work environment itself.

Summary The committee reviewed a number of methods for assessing the effects of the work system on diagnostic error. This section highlights a number of those methods and illustrates how they have been applied in various health care settings to develop insights into the risks of error and to identify potential areas for improvement. The methods have in common the fact that they combine observation of the actual processes (tasks, communication, interaction with technology) with documentation of those processes. These methods can be relatively labor intensive, and they tend to require application at the individual site level, which implies that this is work that all teams and settings in which diagnoses are made need to become more skilled at undertaking. While standardized tools exist (surveys, methods of observation, and analysis of teams) and might be applied to samples of different types of teams and settings to identify particular vulnerabilities for diagnostic error, the most useful application of these methods is typically for improvement at the local level. The human factors science in this area suggests that a number of likely problems can be readily identified—that is, that deep study may not be necessary—but the complexity of the interactions among these various factors suggests that high levels of vigilance and attention to measurement will likely be necessary throughout the health care system.