1.1.3.1. CDS To Generate Diagnoses

1.1.3.2. CDS To Assist With Diagnostic Study Interpretation

Several papers included in this review described investigational studies of CDS tools to assist with diagnostic study interpretation, including imaging studies, electrocardiograms (ECGs), and pathology. Although these CDS tools are proof-of-concept in nature, they demonstrate the potential to augment clinician diagnostic performance but not completely replace it.

1.1.3.3. CDS To Identify Patients at Risk for Diagnostic Errors

Three studies examined the use of CDS tools to identify patients who are at risk of having a diagnostic error.^35,37,38 The systems were all effective at identifying at-risk patients and allowed potential diagnostic errors, including missed or delayed diagnoses, to be prevented, while saving the clinicians time by reducing manual workloads and cognitive burden. As previously discussed, the study by Vandenberghe et al. (2017) used discordance between the diagnoses generated by the AI tool and the diagnoses by the pathologist to flag cases where there may be a high risk of diagnostic error.³⁵

Koopman et al. (2015) developed a system to compare final radiology reports with final ED diagnoses to ensure that the ED identified and appropriately treated an abnormality on radiologic examination. A text analysis system first screens radiology reports to identify limb abnormalities, including fractures, dislocations, and foreign bodies. If the system identifies an abnormality, the diagnosis is reconciled with the ED diagnosis, as defined by International Classification of Diseases, 10^th Revision (ICD-10) codes. If there is a discrepancy, the chart is flagged as a possible misdiagnosis, allowing immediate review and followup. Across the three settings in which the study took place, 274 of 2,018 patients (13.6%) with radiologic abnormalities were flagged for potentially missed diagnoses, and the chart was reviewed manually. Nine of the cases were identified as truly missed diagnoses, and the other instances were due to the ED ICD-10 discharge diagnoses being ambiguous, and not indicative of a diagnostic error. The value in this method is that clinicians need to review only a small subset of the radiology reports, in this case 11 percent of the total number or radiology studies, to determine whether there were potentially missed diagnoses.³⁷

Murphy et al. (2015) applied electronic triggers to EHR data to identify the presences of “red flags,” exclude records for which further evaluation is not warranted (e.g., patients in hospice), and identify the presence of a delay in diagnostic evaluation for three conditions: colon cancer, lung cancer, and prostate cancer. Examples of red flags include positive fecal occult blood testing for colon cancer, concerning imaging studies for lung cancer, and elevated prostate-specific antigen for prostate cancer. Delayed diagnostic evaluation was defined by the absence of documented followup action. The trigger flagged 1,256 patients out of 10,673 patients with abnormal findings (11.8%) as being high risk for delayed diagnostic evaluation. Of these, 749 were true positives, a positive predictive value of 59.6 percent. Times to diagnostic evaluation were significantly lower in intervention patients compared with control patients flagged by the colorectal trigger and prostate trigger. There was no significant difference for the lung trigger.³⁸

1.1.3.4. Unintended Consequences

In general, the CDS tools have an added benefit of improving access to specialized care by providing the clinician with assistance in diagnosing conditions that would typically fall in the realm of a specialist.^12,19,27

Several of the CDS tools identified in this review, in addition to improving diagnostic accuracy, would also allow prioritization of work, creating greater efficiencies and improving workflow once implemented in clinical settings.^27,31,38 These systems flagged studies or diagnoses that required followup, allowing the clinicians to prioritize their work.

For the CDS tools that generate DDX, Graber and Mathew (2008) raised the concern that presenting the clinician with a long list of diagnostic possibilities could be distracting or lead to unnecessary testing and procedures.³ Elkin et al. (2010) suggested that these tools actually reduce the cost of care by assisting the clinician with a broader differential diagnosis list, which is more likely to contain the correct diagnosis. In the case of the DXplain tool, providing the list of diagnoses in order of likelihood can lead to the clinicians evaluating the more likely diagnoses earlier.³⁹

1.1.4.1. Facilitators

Since many of the studies were conducted to validate algorithms or were exploratory in nature (e.g., testing AI algorithms to determine their ability to predict correct diagnosis), few described experiences with implementation in real clinical settings.

In the meta-review of systematic reviews by Nurek et al. (2015), the authors determined the features and effectiveness of computerized diagnostic decision support systems for medical diagnosis in primary care. The authors identified conditions that need to be met if a fully integrated CDS tool for diagnoses is to be successfully implemented and used: the tool can readily be integrated into EHRs; is based on standard terminologies, such as diagnosis codes (e.g., ICD-10); has the ability to be easily updated; is thoughtfully integrated into the clinicians’ cognitive workflow; and interfaces with the clinicians at appropriate action points.⁴⁰

1.1.4.2. Barriers

The information generated by CDS for use in diagnosis is only as good as the information that is put into the system. For example, if the clinician interprets the physical exam incorrectly (e.g., saying that a physical sign is absent when it is present) and inputs that incorrect information into the tool, that error may negatively affect any diagnosis that is partially based on the presence of that sign.^10,11,25,37 In the study by Koopman et al. (2015), discharge diagnoses, as indicated by ICD-10 codes, are reconciled with the diagnosis from radiology reports. If the ICD-10 code is incorrect, the system may not recognize a potential missed diagnosis.³⁷ Gegundez-Fernandez et al. (2017) commented that accurate diagnosis can be achieved only if the clinician’s assessment of the patients’ signs and symptoms is correct, because the automated system will process only data that humans introduce.¹⁰

In the case of ECG interpretation, accurate ECG recording depends on many variables, including lead placement, weight, movement, coexisting electrolyte abnormalities, and symptoms. If the placement is wrong (e.g., leads are placed in wrong location), the interpretation may be wrong.^33,34

1.1.6.1. Leveraging the “CDS Five Rights” Approach

A useful framework for achieving success in CDS design, development, and implementation is the “CDS Five Rights” approach.⁴¹ The CDS Five Rights model states that CDS-supported improvements in desired healthcare outcomes can be achieved if we communicate: (1) the right information: evidence-based, suitable to guide action, pertinent to the circumstance; (2) to the right person: considering all members of the care team, including clinicians, patients, and their caretakers; (3) in the right CDS intervention format, such as an alert, order set, or reference information to answer a clinical question; (4) through the right channel: for example, a clinical information system such as the EHR, a personal health record, or a more general channel such as the Internet or a mobile device; (5) at the right time in workflow, for example, at the time of decision/action/need. CDS has not reached its full potential in driving care transformation, in part because opportunities to optimize each of the five rights have not been fully explored and cultivated.⁴²

Providing the Right Information to the End User: The process of integrating real-time analytics into clinical workflow represents a shift towards more agile and collaborative infrastructure building, expected to be a key feature of future health information technology strategies. As interoperability and big data analytics capabilities become increasingly central to crafting the healthcare information systems of the future, the need to address issues that ease the flow of health information and communication becomes even more important. Without tools that select, aggregate, and visualize relevant information among the vast display of information competing for visual processing, clinicians must rely on cues by “hunting and gathering” in the EHR. Alerts that embody “right information” should provide just enough data to drive end user action, but not so much as to cause overload.⁴³ Overload can create alert fatigue and lead to desensitization to the alerts, resulting in the failure to respond to warnings, both important and less important. Experience from the use of CDS in the medication ordering process has demonstrated this paradoxical increase in risk of harm due to alerts that were intended to improve safety.^44,45

Providing Information in the Right Format: Lack of knowledge regarding how to present CDS to providers has impeded alert optimization, specifically the most effective ways to differentiate alerts, highlighting important pieces of information without adding noise, to create a universal standard. The potential solution that CDS represents is limited by problems associated with improper design, implementation, and local customization. In the absence of evidence-based guidelines specific to EHR alerting, effective alert design can be informed by several guidelines for design, implementation, and reengineering that help providers take the correct action at the correct time in response to recognition of the patient’s condition.⁴⁶

Right Workflow: A well-thought-out user-centered design or equivalent process during the implementation phase includes critical elements of leadership buy-in, dissemination plans, and outcome measurements. Knowledge needs to be gained about how to implement the CDS and how to create an interface between the system and the clinician that takes into consideration the cognitive and clinical workflow.^27,47 The optimal approach to CDS should not be focused primarily—or even secondarily—on technology. Implementation is about people, processes, and technology. Systems engineering approaches, including consideration of user experience and improvements in user interface, can greatly improve the ability of CDS tools to reach their potential to improve quality of care and patient outcomes. The application of human factors engineering in determining the right workflow includes but is not limited to ethnographic research including workflow analysis and usability testing.

1.1.6.2. Trust in Automation

CDS is meant to augment clinician performance, not replace it, making it an imperative to carry existing work forward into actual clinical settings.¹ CDS has advanced to the point of becoming a “type of automation that supplements the human powers of observation and decision.” Technologies related to big data bring both exciting opportunities and worrying prospects for misinformation, disinformation, and falsified information. Further work is required to demonstrate clinical and economic evidence using data from a population representative of the health system in a way that clinicians find trustworthy.

1.1.6.3. Measurement

Successful CDS deployment requires evaluating not only whether the intended clinicians are using the tool at the point of care, but also whether CDS use translates into improvements in clinical outcomes, workflows, and provider and patient satisfaction. However, success measures are often not clearly enunciated at the outset when developing or implementing CDS tools. As a result, it is often difficult to quantify the extent to which CDS has been effectively deployed, as well as whether it is effective at managing the original diagnostic problem it was designed to address.

1.2.3.1. Use of RNS for Radiologic Studies

Five studies focused on the impact of RNS on the communication of CSTR in radiology. The CSTR ranged from results requiring treatment but not immediately life-threatening to immediately life-threatening results. The impact of the RNS on the communication of results, and action taken on the results, was mixed.

Two studies, both by Lacson and colleagues, evaluated the use of an Alert Notification of Critical Results (ANCR) system to facilitate communication of critical imaging test results to ordering clinicians at a large academic medical center. The ANCR system, integrated into the clinical workflow, allows both synchronous communication (e.g., pagers) for results related to life-threatening conditions, and asynchronous communications (e.g., email). The system relies on radiologists who read and interpret the radiographic images to initiate an alert to the ordering clinician, rather than using a completely automated system. In the first study, the authors evaluated the ANCR system on adherence to a hospital policy for timeliness of notifications that is based on criticality of the imaging result.¹⁵ Using a pre/post study design, the authors found a significant improvement in adherence to the timeliness policy, with adherence increasing from 91.3 percent before the ANCR intervention to 95.0 percent after (p<0.0001). In the second study, also using a pre/post study design, the authors evaluated the impact of implementing both the ANCR system and the policy of communication of the critical imaging test result in reducing critical results that lacked documented communication (date, time, and name of ordering clinician contacted). After the implementation of the critical imaging test result policy and the ANCR, critical results lacking documented communication decreased nearly fourfold between 2009 and 2014 (0.19 to 0.05, p<0.0001).¹⁶

Table 1.1

Overview of Single Studies.

In a study linked to the work of Lacson and colleagues, O’Connor et al. (2015) integrated an ANCR with an EHR-based results management application and evaluated its adoption and impact on followup of actionable results by primary care providers (PCPs) in the outpatient setting. Prior to integration, PCPs used the EHR application to track and acknowledge results from laboratory studies. The integration of the two systems allowed the PCPs to receive and acknowledge the ANCR-generated non-urgent CSTR alerts in the EHR or through the ANCR system. During the 2 years after implementation, 15.5 percent of the ANCR alerts were acknowledged in the EHR (15.6% year 1, 15.4% year 2). In the post-intervention period, there was a significant difference (p=.03) between the proportion of alerts acted upon that were acknowledged in the EHR application (79%; 95% CI, 52 to 92) compared with the alerts acknowledged in the ANCR system (97%; 95% CI, 90 to 99).¹⁷

Singh et al. (2009) evaluated the impact of an EHR-based system to alert clinicians to critical imaging results in a multidisciplinary ambulatory clinic at a large Veterans Administration (VA) medical center and its five satellite clinics. The VA EHR has an embedded notification system for alerting clinicians to CSTR in a “View Alert” window. The system requires that the radiologist reading an image flag abnormal imaging results, and these alerts are then transmitted to the “View Alert” window. During the study period there were 1,196 abnormal imaging alerts generated (0.97% of all imaging studies), and 217 (18.1%) of these alerts remained unacknowledged (i.e., the ordering clinician did not click on and open the alert) after 2 weeks. Using logistic regression, variables associated with a lack of acknowledgement included physician assistants compared with attending physicians (odds ratio [OR]: 0.46; 95% CI, 0.22 to 0.98); resident physicians compared with attending physicians (OR: 5.58; 95% CI, 2.86 to 10.89); and dual communication (i.e., communication with two clinicians) compared with communication with a single clinician (OR: 2.02; 95% CI, 1.22 to 3.36). Notably, 92 alerts, both acknowledged (n=71) and unacknowledged (n=21), lacked followup at 4 weeks.¹⁸

Eisenberg et al. (2010) evaluated the use of a Web-based electronic messaging system to communicate non-urgent CSTRs and recommend followup to ordering clinicians. As in the system used in the studies by Lacson and colleagues, the alerts were initiated by radiologists responsible for interpreting images through a web-based application. The request is received by a facilitator, who is then responsible for conveying the results to the ordering clinician. Once the results have been conveyed, the facilitator sends a confirmation back to the radiologist to close the loop. The authors recognized that the study design was weak (post-only with a satisfaction survey). They authors found that 82.2 percent of the alerts were communicated to the ordering clinicians within a 48-hour window, as defined by the time the radiologist submits a communication request to the time the facilitator conveys the communication to the ordering physician. The authors also found that the day of week affected the outcome, with more alerts submitted by the radiologists Monday–Thursday before 3 p.m. communicated within 48 hours (93.7% +/− 2.4), compared with alerts generated on Thursday afternoon through Sunday (73.0% +/− 9.2). The authors incidentally noted that for one-third of communications in which additional imaging or followup had been recommended, the electronic medical record had no documentation that these services were actually performed.¹⁹

1.2.3.2. Use of RNS for Laboratory Studies

Nine of the included studies focused on the use of RNS for laboratory studies, including one meta-analysis and one systematic review. As was the case for the RNS for radiologic studies, the evaluated interventions varied across studies and included paging, email, text messages, and EHR alerts. Results of the RNS were mixed.

The meta-analysis and the systematic review examined the effectiveness of automated electronic RNS to alert ordering clinicians to CSTR, and found insufficient/inconclusive evidence for the use of these systems.^8,9 The systematic review by Liebow et al. included four studies, two of which were used to calculate a standardized effect size (ES).^10,20 Etchells et al. reported results of a randomized controlled trial evaluating an automated RNS that sends critical laboratory values directly from the laboratory information system to a pager carried by the ordering physician. The objective was to evaluate the effect of the system on physician response time, defined as the time from when the critical result is entered into the lab system to the time an order is written in response to the critical value, or the documented time of treatment (whichever is relevant). They found a 23-minute reduction in median response time, 16 minutes (interquartile range [IQR] 2–141) for the automated paging group, and 39.5 minutes (IQR 7–104.5) for the usual care group, but this difference was not statistically significant.¹⁰ Park et al. used a pre/post design to test the impact of a short message service and callbacks for action on critical hyperkalemia results. Across all patients in both intensive care units and general wards, the median and interquartile ranges for the clinical response times, defined as the frequency of clinical responses divided by the number of critical value alerts during a given time period, were significantly reduced, going from 213.0 minutes in the intensive care unit and 476.0 minutes in the general wards to 74.5 minutes and 241 minutes, respectively (p<.001).²⁰ Using Cohen’s d, Liebow and colleagues calculated a grand mean reduction of time to communicate critical results for these two studies (d=.42; 95% CI, 0.23 to 0.62), indicating that the time to report a randomly selected CSTR using the automated system will be shorter than with a randomly selected manually reported value 61.8 percent of the time. Liebow et al. gave an overall strength of evidence rating of “suggestive” for automated RNS.⁸

Liebow et al. also conducted a systematic review of five studies evaluating the use of centralized call centers that communicate critical CSTRs to the ordering clinician.⁸ Four of the five studies whose primary outcome was percent of calls completed within a specified interval after results were available from the laboratory (either <30 min or <60 min) contained sufficient data to calculate a standardized ES. The results of a random-effects meta-analysis support the implementation of call centers (mean OR=22.2; 95% CI, 17.1 to 28.7). This translates to critical lab values being reported faster with the call system than results reported via usual means (e.g., call to unit by laboratory technologist) approximately 88.6 percent of the time. Liebow et al. consider the overall strength of evidence for call center systems to be “moderate.”

A systematic review by Slovis et al. included 34 articles published through 2016, representing 40 years of research related to asynchronous automated electronic laboratory RNS.⁹ Although a wide variety of systems were represented and the study designs and outcomes differed, the authors summarized that these systems can be successfully implemented and improve timeliness of result notification and action. On closer examination of the five most recent studies that were included in the review and also identified through our search, the findings neither fully supported nor opposed use of these systems.^{10,11,20–22}

In the first of two randomized controlled trials by Etchells et al. (2010) and included in the systematic review by Slovis et al., an automated paging system to convey critical laboratory results was evaluated in an urban academic medical center.¹⁰ As described above, although there was a 23-minute reduction in the median response time, this was not statistically significant. In their second study, Etchells et al. (2011) combined an automated RNS with CDS.¹¹ The alerts, sent via text to a smart phone or to a pager, contained information about the specific patient and the abnormal result, and offered a URL to a webpage with decision support for the specific alert. The primary outcome was the proportion of pre-defined potential actions that were completed in response to the alert. A secondary outcome was the number of adverse events, defined as worsening of the patient’s condition or complications related to the treatment of the condition. The median proportion of potential clinical actions that were completed was 50 percent (IQR 33–75%) with the alerting RNS with CDS and 50 percent (IQR 33–100%) without it, a difference that was not statistically significant, Without the system, there were 111 adverse events (33%) within 48 hours following an alert and with the alerting system on, there were 67 adverse events (42%); a 9 percent increase when using the alerting system that bordered on statistical significance (p=0.06).

In addition to the five studies included in Slovis et al, two additional studies about laboratory RNS were reviewed for this report. In the outpatient department of a large (2,500-bed) tertiary teaching hospital in Taiwan, Lin et al. studied the impact of a phone-based RNS on clinical outcomes of patients taking the anticoagulant warfarin.²³ Their RNS automatically generates and delivers text messages about critical lab CSTRs to providers’ mobile phones 24 hours a day, 7 days a week. Using a pre/post study design, the investigators found no significant differences in warfarin-associated adverse events. The rate of major venous thromboembolism events was 1.6 percent for both the manual alert period and the test RNS period. The rate of major hemorrhage requiring an ED visit or hospital admission was 3.1 percent in the manual alert period and 4.2 percent in the RNS alert period (p=0.198). As with the findings in Etchells et al. (2010), the secondary outcome of timeliness of physician followup actions after receipt of an automated critical alert was not significantly improved (11.13 ± 7.65 days for manual alert period vs. 11.32 ± 8.17 days for phone-based RNS period; p=0.814).

Expanding on the work previously described by Lacson and O’Connor, O’Connor et al. (2018) examined the use of the ANCR for communication of non-urgent clinically significant pathology reports indicating new malignancies.²⁴ After a pathologist identifies the CSTR, the ordering physician is contacted via pager about critical or urgent results, and via pager or secure email for non-urgent results, and the CSTR is entered into the ANCR system. For results that the ordering physician does not acknowledge, the system sends reminders to the pathologist and the ordering physician. Acknowledgment of the CTSR within 15 days, the institutional policy for non-urgent CSTR, was documented for 98 of 107 cases (91.6%) before the RNS had been implemented, and for 89 of 103 (86.4%) after the RNS had been implemented, a difference that was not statistically significant. There was also no significant difference in median time to acknowledgment for new malignancies when comparing the pre-RNS period (7 days; IQR 3–11) and post-intervention period (6 days; IQR 2–10). In the post-RNS period, for CTSR using the ANCR, median time to acknowledgment was significantly shorter than when an ANCR alert was not generated (2 vs. 7 days, p=0.0351).

1.2.3.3. Use of RNS for Tests Pending at Discharge

Tests pending at discharge (TPADs) involve transitions, span more than one setting (e.g., the hospital setting to the ambulatory setting), and often involve more than one clinician (e.g., inpatient attending physician and outpatient primary care physician). The risk of missed communication and potential harm to patients is greater during these transitions between settings and clinicians.^25,26 Three cluster-randomized controlled studies from a single institution investigated the use of an automated email CSTR notification system for TPAD.^12–14 Awareness and confirmed acknowledgement of the test result after discharge were statistically higher in the intervention group, but there was no difference between the intervention and control groups in documented actions taken in response to the test results (i.e., receiving/confirming receipt of a test result did not improve timeliness of acting upon that information).

Two cluster-randomized controlled studies by Dalal et al. (2014 and 2018) included inpatient attending physicians and PCPs whose patients were discharged from inpatient cardiology and medicine units in a large academic medical center, and who had TPAD for both radiology and laboratory studies.^12,13 In these studies, a patient’s discharge triggers a series of electronic events that updates the status of any remaining TPADs on a daily basis. As results for these pending tests are finalized, the responsible in-patient attending and outpatient PCP receive an automatic email containing the test result. The primary outcome of the first study was self-reported awareness of the TPAD result by the patient’s inpatient attending physician.¹² There was a statistically significant increase in the awareness of TPAD results by attending physicians for patients assigned to the intervention compared with those assigned to usual care (76% vs. 38%, adjusted/clustered OR 6.30, 95% CI, 3.02 to 13.16, p<0.001). The second study was larger, and the primary outcome was the proportion of actionable TPADs with documented action in the EHR.¹³ For the primary outcome of documentation of action, there was no significant difference between the intervention and usual care groups (60.7% vs. 56.3%). For those that had an action documented, the median days between result notification and documented action was significantly lower in the intervention group (9 days, CI, 6.2 to 11.8) compared with the usual care group (14 days, 95% CI, 10.2 to 17.8) (p=0.04).

In the third study, by El-Kareh et al., the automated RNS described previously was used to alert inpatient and outpatient physicians about positive cultures when the final lab result was returned after patient discharge and the patient was not adequately treated with antibiotics. The alerts included patient identifiers, names and contact information for the physicians involved in their care, the culture results, the discharge medication list, and patient allergy information. Twenty-eight percent of results in the intervention group and 13 percent in the control group met the primary outcome of documented followup (in outpatient chart) within 3 days of receipt of the post-discharge lab result, a statistically significant difference [adjusted OR 3.2, 95% CI, 1.3 to 8.4; p=0.01].²⁷

1.2.3.4. Unintended Consequences

Study authors raised a hypothetical concern about alert fatigue, a potential unintended consequence of implementing alerting RNSs, but only one study measured a related outcome: overuse of the alerting system. Lacson et al. (2016) found that the proportion of reports without critical CSTR and using the ANCR was significantly less than when the ANCR was not used (0.09 vs. 0.20, p<0.002, χ2 test).¹⁶ Etchells et al. (2010) noted that critical results, such as those from repeated troponin tests, were viewed as nuisances by receiving clinicians during a pilot of the system.¹⁰ They also noted that because physician schedules were not fully automated, it was not possible to consistently route critical results to a responsible and available physician to take action. To compensate for this, physicians handed off “critical value pagers” so that the physician-on-call carried several pagers. Although this could reduce the number of missed alerts, it also created confusion when the on-call physician often could not discern which pager was alerting.

Unexpectedly, Singh et al. (2009) found that dual communication, a duplication intended to ensure that at least one physician received the alert, was associated with delayed followup. This finding was attributed to the lack of clarity about who was responsible for handling the alert.¹⁸

1.2.4.1. Facilitators

1.2.4.2. Barriers

1.3.3.1. General Training in Clinical Reasoning

Clinical reasoning is the process by which clinicians collect data, process the information, and develop a problem representation, leading to the generation and testing of a hypothesis to eventually arrive at a diagnosis.^5,6

Cook et al. (2010) conducted a meta-analysis and systematic review of the effects on training outcomes of using virtual patients, including the effects on clinical reasoning. The learners interact with a computer program that simulates real-life clinical scenarios to obtain a history, conduct a physical exam, and make diagnostic and treatment decisions. In comparing virtual patients to no intervention, the pooled ES for the five studies with an outcome of clinical reasoning was 0.80 (95% CI, 0.52 to 1.08). Pooled ESs for the outcomes of knowledge (N=11 studies) and other skills (N=9 studies) were also large. When comparing the use of virtual patients to noncomputer instruction (e.g., didactic instruction, standardize patients, routine clinical activities), the pooled ES for the outcome of clinical reasoning was −0.004 (95% CI, −0.30 to 0.29, N=10 studies), and it was also low for satisfaction, knowledge, and other skills. The main takeaway from this meta-analysis and review was that the use of virtual patients is associated with large positive effects on clinical reasoning and other learning outcomes when compared with no intervention and is associated with small effects in comparison with noncomputer instruction.⁷

Graber et al. (2012) in their systematic review identified papers that reported testing interventions aimed at reducing cognitive errors.⁸ Three broad categories of interventions emerged: interventions to improve knowledge and experience, interventions to improve clinical reasoning, and interventions that provide cognitive support. Several papers examined the use of training in metacognitive skills to improve clinical reasoning, which is below. Wolpaw et al. (2009) studied the use of a learner-centered technique by third-year medical students to present clinical cases in a structured manner (SNAPPS: Summarize, Narrow, Analyze, Probe, Plan, Select). Although the authors did not assess whether the DDX were accurate, they found that students using the SNAPPS technique performed better on all outcomes, including analyzing possibilities of the DDX, expressing uncertainties, and obtaining clarification.⁹

1.3.3.2. Training in Metacognitive Skills To Reduce Biases

Cognitive biases can affect clinical reasoning and influence the diagnostic process, contributing to a large proportion of misdiagnoses.^6,8,10,11 Metacognition, the understanding, control, and monitoring of one’s cognitive processes, can be used to gain better insight and counteract these biases.^12,13 Nine studies focused on techniques to enhance metacognitive skills, specifically training on the use of cognitive forcing strategies (CFS) and the use of reflection during the diagnostic process. The results of the studies are mixed, but overall suggest the use of training metacognitive strategies to improve diagnostic performance.

The use of CFS, a metacognitive strategy, is a technique to bring about self-monitoring of decision making and to force the consideration of alterative diagnoses.¹³ Three studies (Sherbino et al., 2011, Sherbino et al., 2014, Smith and Slack, 2015) provided medical students and residents with training on the use of CFS and measured its impact on diagnosis.^14–16 The results did not show any appreciable improvement in diagnostic accuracy. In a preliminary and followup study, Sherbino et al. employed a 90-minute, standardized, interactive, case-based teaching seminar on CFS for medical students during their emergency medicine rotation.^14,15 Neither study showed any improvement in diagnostic errors. In the first study, they found that fewer than half of the students could use the CFS to debias themselves, and that 2 weeks post-training the students’ knowledge of debiasing was no longer present. In the second study, there was no difference in the diagnostic accuracy between the control and intervention groups. In the study by Smith and Slack (2015), family medicine residents participated in a debiasing workshop that included training on CFS. They found that the residents’ ability to formulate an acceptable plan to mitigate the effect of cognitive biases significantly improved after the training (p=0.02), although the residents were not able to translate the plan into practice, as evidenced by no change in the outcomes of preceptor concurrence with the residents’ diagnoses, residents’ ability to recognize their risk of bias, and the preceptors’ perception of an unrecognized bias in the residents’ presentations. This study was limited in that CFS targets biases related to nonanalytic reasoning (so-called pattern recognition). Novice diagnosticians, such as medical students, may lack sufficient experience to employ nonanalytic reasoning, rendering these methods increasingly more useful as experience increases.¹⁶

In a frequently cited study, Mamede et al. (2010) investigated whether recent diagnostic experiences elicit availability bias (i.e., judging a diagnosis that comes to mind more readily as being correct), and then tested a simple instructional procedure to reduce that bias. The training consisted of a five-step procedure to induce structured reflection and improve diagnostic accuracy in first- and second-year internal medicine residents. The use of reflection did reduce availability bias and improved diagnostic accuracy.¹⁷ Two additional studies by this group of investigators (Mamede 2012, Mamede 2014), furthered this work on the use of structured reflection as a tool to facilitate diagnosis. The first of these studies found that the use of structured reflection after providing an immediate diagnosis when practicing with clinical cases fostered the learning of clinical knowledge more effectively than providing an immediate diagnosis only, or generating an immediate diagnosis followed by a differential diagnosis.¹⁸ In the second study, the use of this technique enhanced learning of the diagnosis practiced as well as its alternative diagnoses.¹⁹

Coderre et al. (2010) tested the effectiveness of questioning a medical student’s initial hypothesis as a means to induce cognitive reflection. The authors found that the questioning of an initial correct diagnosis did not change the final diagnosis; the students tended to retain the initial diagnosis. If the student’s initial diagnosis was incorrect, the questioning provided an opportunity for the students to recognize and react to their error, and correct their diagnosis.²⁰

In a randomized controlled study, Nendaz et al. (2011) studied the impact of weekly in-person case-based clinical reasoning seminars incorporating diagnostic reflection, during which the students were prompted to reflect on their reasoning process and were provided feedback on each step of that process. They found no difference in the accuracy of the medical students’ final diagnoses between intervention and control groups (74% vs. 63%), although the students in the intervention group were more likely to have mentioned the correct diagnosis somewhere on their working list of DDX under consideration (75% vs. 97%, p=0.02).²¹

Reilly et al. (2013) incorporated the promotion of reflection on past experiences where a cognitive bias led to a diagnostic error, as part of a longitudinal curriculum on cognitive bias and diagnostic errors for residents. Residents who completed the curriculum significantly improved their ability to recognize cognitive biases when compared with their baseline performance (p=0.002) and when compared with the control group (p<0.0001). The study was limited in that it did not evaluate the impact of the intervention on diagnostic accuracy.²²

1.3.3.3. Training on the Use of Heuristics

Heuristics are decision strategies, or mental shortcuts, that allow fast processing of information to arrive at a decision or judgment. One type of heuristic is representativeness; the use of the degree to which an event is representative of other, similar events to assess the probability of an event occurring.^23,24 Although the literature around the use of heuristics in medicine tends to focus on the biases they introduce, there is a recognized potential for training with heuristics to achieve better diagnostic accuracy.^25,26

Mohan et al. (2018) conducted a randomized controlled trial comparing two training interventions designed to improve the use of the representativeness heuristic to improve trauma triage by emergency physicians. The authors developed two serious video games to train in the use of the heuristic. The first was an adventure game, based on the theory of narrative engagement, and the second was a puzzle-based game, based on the theory of analogical reasoning, using comparisons to help train the learners on applying decision principles. Both games incorporated feedback on diagnostic errors and how they could be corrected. Results showed that both games had positive effects on trauma triage, whereas traditional medical education had none.²⁶

1.3.3.4. Training To Improve Visual Perception Skills

In radiology, diagnostic errors fall into two broad categories: perceptual errors, in which an abnormality on an image is not seen or identified, and interpretive errors, in which an abnormality is seen but the meaning or the importance of the finding is not correctly understood.^27,28 Perceptual errors account for a majority of misdiagnoses in radiology,^27,29,30 and can be rooted in faulty visual processing or, to a lesser extent, cognitive biases.³¹

Improving visual perception skills, which predominate the diagnostic process in radiology, requires methods of training different from those to improve clinical reasoning.²⁸ Four studies were identified through our search that evaluated the impact of educational interventions on perceptive skills, with three showing improvement in perceptive performance.^32–34 The studies involved subjects early in their medical training, and each tested a different intervention to improve perceptive performance, making aggregation of findings challenging.

A novel study by Goodman and Kelleher (2017) took 15 first-year radiology residents to an art gallery, where experts with experience in teaching fine art perception trained the residents on how to thoroughly analyze a painting. The trainees were instructed to write down everything they could see in the painting, after which the art instructor showed the trainees how to identify additional items in the painting that they had not perceived. To test this intervention, the residents were given 15 radiographs pre-intervention and another 15 post-intervention and asked to identify the abnormalities. At baseline, the residents scored an average of 2.3 out of a maximum score of 15 (standard deviation [SD] 1.4, range 0–4). After the art training, the residents’ scores significantly improved, with an average score of 6.3 (SD of 1.8, range 3–9, p<.0001), indicating that perception training may improve radiology residents’ abilities to identify abnormalities in radiographs.³²

In a small randomized crossover study by van der Gijp et al. (2017), 19 first- and second-year radiology residents received training on two different visual search strategies to determine their effect on accuracy of detecting lung nodules on CT scans. The first search strategy was “scanning,” in which the resident views all the visible lung tissue on a single image and slowly scrolls down, image by image, through the entire study. The second search strategy, “drilling,” has the resident mentally divide each lung into three regions and scroll through each region individually while keeping the eyes fixed on that region. Perceptual performance for both scanning and drilling strategies and a pre-test using a free search strategy was determined by the mean numbers of true positives and false positives. There was a significant effect of year of residency on the true positive score (p<0.01) but not for false positives. Drilling (p<0.001) and free search (p<0.001) both resulted in significantly higher true positive scores (i.e., lung nodules identified appropriately) than did scanning. There was no difference between the drilling or free search strategies. The free search strategy resulted in higher false positive scores than drilling (p<0.001) and scanning (p<0.001). The authors concluded that drilling outperforms scanning for detecting lung-nodules and should be the preferred strategy when teaching perceptive skills.³³

In a randomized controlled trial, Soh et al. (2013) used an online e-learning tutorial to train 14 first-year medical radiation sciences students (i.e., radiology technologists) in Australia to improve their ability to detect breast lesions on mammographic images. The 1-hour tutorial focused on anatomy, image positioning, mammogram viewing and analysis, and the appearance of normal breast tissue and asymmetric densities and masses. The students were randomized to the intervention (tutorial) or control group, with their performance evaluated by their viewing normal and abnormal mammograms. The study used eye-tracking technology to determine when and how often the student fixated on a lesion. The intervention group demonstrated improvement in the mean number of fixations per case (p=.047), and decreased time to first fixation on a lesion by 49 percent (p=.016). The intervention increased students’ ability to identify lesions (i.e., sensitivity) by 30 percent (p=.022).³⁴

The fourth study evaluated different proportions of normal and abnormal radiographs in image training sets to determine the best case-mix for achieving higher perceptive performance.³⁵ For the intervention, Pusic et al. (2012) used three different 50-case training sets, which varied in their proportions of abnormal cases (30%, 50%, 70%). One hundred emergency medicine residents, pediatric residents, and pediatric emergency medicine fellows were randomized to use one of the training sets. After the intervention, all participants completed the same post-test. All three groups showed improvement after the intervention, but with varying sensitivity-specificity trade-offs. The group that received the lowest proportion (30%) of abnormal radiographs had a higher specificity and was more accurate with negative radiographs. The group that trained on the set with the highest proportion of abnormal radiographs (70%) detected more abnormalities when abnormalities were present, achieving higher sensitivity. These findings have significant implications for medical education, as it may be that case mix should be adjusted based on the desired sensitivity or specificity for a given examination type (e.g., screening exams vs. diagnostic test).³⁵

1.3.3.5. Other Education and Training Interventions

In the systematic review of patient safety strategies targeted at diagnostic errors, McDonald et al. (2013) identified 11 studies, ranging in dates from 1981 to 2011, that involved a variety of interventions. Two randomized trials targeted patients and families, and found that the interventions improved performance: the first found that parent education improved parents’ ability to identify serious symptoms requiring a physician office visit, and the second showed that patient education, in addition to reminders, improved breast cancer screening rates.³⁶

Medical teaching typically involves error avoidance training (EAT), in which the focus is on how to perform a task correctly rather than on how to manage errors. However, evidence suggests that in the early stages of learning a skill, errors are necessary in order to avoid them in the future.³⁷ In their randomized trial with 56 medical students, Dyre et al. (2017) compared the use of error management training (EMT), in which students were given “permission” to make errors while conducting simulation-based ultrasound examinations, with EAT. Two outcomes were measured: the objective structured assessment of ultrasound scale, a measure of ultrasound performance; and diagnostic accuracy. The scale scores showed a significant improvement by the EMT group compared with the EAT group (p<0.001). Although diagnostic accuracy showed some improvement, this was not statistically significant. The study is limited in that the authors cannot determine whether the outcomes were a result of EMT, the positive framing of errors, or the combination of these two components.³⁸

In a quasi-randomized controlled trial, Schwartz et al. (2010) tested the use of in-person didactic sessions to teach fourth-year medical students skills in contextualizing patient care. The authors based this work on the premise that there are both biomedical and contextual data that must be ascertained during the diagnostic process and incorporated into the treatment plan. Students who participated in the didactic sessions were significantly more likely to probe for contextual issues and significantly more likely to develop appropriate treatment plans for the standardized patients with contextual issues.³⁹

Two studies investigated the use of online training to improve specific diagnostic skills, both resulting in significant improvements in diagnostic accuracy.^40,41 Smith et al. (2009) conducted a 4-month online didactic continuing education program to improve the ability of radiographers in rural areas to interpret plain musculoskeletal radiographs. The results showed a significant improvement in image interpretation accuracy for more complex cases (p<0.05), although there was no change in accuracy for less complex cases.⁴⁰ McFadden et al. (2016), using convenience sampling, compared a traditional in-person training with an on-line simulation-based training, both designed to improve the diagnosis by primary care practitioners of the etiology of joint pain. The online training included interactive practice opportunities and feedback delivered by an AI-driven tutor. The intervention group’s diagnostic performance was significantly improved from baseline (p<0.02) compared with the group that received the traditional training.⁴¹

1.3.4.1. Facilitators

Several of the studies used interventions that brought the education to the learner, such as those using information technology–based platforms.^34,35,40,41 Although there are costs in both time and money associated with the set-up of these online learning systems, once implemented, the training is more easily administered to more learners and is more flexible to being customized.⁴²

1.3.4.2. Barriers

The use of cognitive training interventions, such as reflective practice, may yield the greatest improvements for only the most complex diagnostic cases.^17,21 This makes application of appropriate strategies in actual clinical settings difficult, as whether a case is complex is often not determined until after the diagnostic process has begun.

In addition, some of these teaching techniques, such as those using standardized patients or requiring development of simulations, are labor intensive and may not be generalizable.

1.4.3.1. Traditional Peer Review: Random Versus Nonrandom Selection

Evaluation of professional practice, which can be accomplished through peer review, is a requirement for accreditation by organizations such as the American College of Radiology (ACR) and The Joint Commission, and recommended by professional associations such as the College of American Pathologists. The best-known example is that used in radiology, the ACR’s RADPEER™ program, which is a standardized process with a set number of cases targeted for review (typically 5%) and a uniform scoring system.⁹ The cases, which are originally interpreted images being used for comparison during a subsequent imaging exam by the reviewing “peer” radiologist, are randomly selected and scored.^5,6 Scores are assigned based on the clinical significance of the discrepancy between the initial radiologist’s interpretation and the review radiologist’s interpretation: (1) concur with interpretation; (2) discrepancy in interpretation, correct interpretation is not ordinarily expected to be made (i.e., an understandable miss); and (3) discrepancy in interpretation and the correct interpretation should be made most of the time. Scores of 2 and 3 can be modified with an additional designation of (a) unlikely to be clinically significant or (b) likely to be clinically significant. Scores of 2b, 3a, or 3b are reviewed by a third party, typically a department chair, medical director, or quality assurance committee.⁹ Discrepancy rates can then be calculated for individual radiologists and used for comparison against peer groups or national benchmarks, and for improving practice.

Six studies involved the use of random peer review strategies similar to that of RADPEER. Each of these studies calculated discrepancy rates of case interpretation between the initial physician’s diagnosis and the peer reviewer’s diagnosis, with some studies using a third party to adjudicate the presence and severity of a diagnostic error.^10–13 Four of the studies compared random versus nonrandom approaches.^11–14 Table 1.2 lists the studies by case type, case selection, and discrepancy (i.e., diagnostic error) rates. The definition of discrepancy varied slightly across studies, but typically ranged from minor disagreements between reviewers that would necessitate a change to a report but are incidental to treatment, to major disagreements that may directly affect a patient’s outcome.

Table 1.2

Discrepancy Rates for Peer Review in Pathology and Radiology.

Cases selected for review by a random process consistently had lower discrepancy rates between the initial interpretation and the peer review interpretation (0.8%–3.8%) than the cases selected nonrandomly (1.6%–13.2%). The more focused the case selection criteria, the higher the yield of identified diagnostic errors.

In their study of the effectiveness of random and focused reviews in anatomic pathology, Raab et al. found that the 5 percent targeted random selection method identified significantly fewer cases with diagnostic errors than the focused review of case types known to be diagnostically challenging or where there is a lack of standardization (2.6% vs. 13.2%, p<.001).¹² In practical terms, the focused review detected approximately 4 times the number of errors compared with the random reviews, which involved 20 times the number of specimens. Layfield and Frazier compared four different methods of anatomic pathology case selection and found that randomly targeted cases had the lowest rate of identified diagnostic errors (0.8%) compared with the three non-random methods. The focused review, in which all cases of a specific type are reviewed (all dermatopathology cases), identified the greatest percentage of diagnostic errors (8.5%).¹⁴ In a study by Itri et al., at an institution where radiologists are required to review 20 randomly selected cases per month, the discrepancy rate was found to be 2.6 percent, with all identified errors being considered minor discrepancies.¹¹ The authors also found that, among 190 additional cases selected for review because of concern about potential errors, 130 (68.4%) had significant discrepancies: 94 were significant discrepancies that may affect treatment but not outcomes, and 36 were major discrepancies that may affect outcomes. In a study conducted at a large, urban, multidisciplinary children’s hospital, Swanson et al. (2012) describe discrepancy rates of 3.6 percent using their mandatory random peer review process, where each radiologist reviews four cases per shift. Radiologists could also conduct nonrandom reviews on cases of interest or concern, or if they were referenced in a clinical consultation or part of a review conference. There was a 12-percent discrepancy rate using this method.¹³

One study combined the use of random case selection with a prospective review approach, where the peer review occurred prior to report finalization.¹⁵ The rate of discrepancies using this method was 2.6 percent, aligning with the findings of the other studies using random case review. There was one discrepancy considered to be of major significance identified, which was resolved prior to patient care decisions being made.

1.4.3.2. Double Reading

A common form of nonrandom peer review, particularly in radiology practice, is the use of double reading, in which a second clinician reviews a recently completed case.⁵ With this method the review is integrated into the diagnostic process rather than conduced retrospectively, allowing errors to be identified and resolved prior to a report being transmitted to the ordering provider or the patient.

Geijer and Geijer (2018) reviewed 46 studies to identify the value of double reading in radiology. The studies fell into two categories: those that used two radiologists of similar degree of subspecialization (e.g., both neuroradiologists) and those that used a subspecialized radiologist only for the second review (e.g., general radiologist followed by hepatobiliary radiologist). Across both types of studies included in the review, double reading increased sensitivity at the expense of reduced specificity. In other words, double reading tended to identify more disease, while also identifying disease in cases that were actually negative (i.e., false positives). With discrepancy rates in studies between 26 and 37 percent, the authors suggest that double reading might be most impactful for trauma CT scans, for which there are a large number of images generated that need to be read quickly under stressful circumstances. The authors also suggest that it may be more efficient to use a single subspecialized radiologist rather than implement double reading, as using a subspecialist as a second reviewer introduced discrepancy rates up to 50 percent. This was thought to be a result of the subspecialist changing the initial reports and the bias introduced by having the subspecialist being the reference standard for the study.¹⁶

Pow et al. (2016) reviewed 41 studies to assess the use of double reading on diagnostic efficacy for screening and diagnostic imaging studies. As with the previously described systematic review, the use of double reading was found to increase sensitivity and reduce specificity, making it more desirable for tests, such as cancer screening, where high sensitivity is desired.¹⁷ Also consistent with Geijer and Geijer (2018), the authors recommended the use of double reading in trauma due to the large number of images generated and emergent need for results. They also found that the level of expertise of the reviewers influences the error rate, with those review processes using a subspecialist for the second review having higher rates of error detection than those using two radiologists with similar training.

Four studies evaluated the use of double reading, in which the second reading occurred either concurrently with or in immediate proximity to the first reading.^18–21 In each of these studies, significant numbers of discrepancies were determined to be clinically significant by RADPEER scoring criteria. In Agrawal et al, dual reporting identified 145 errors (3.8%; 95% CI, 3.2 to 4.4) that led to report modification, with 69 determined to be clinically significant.¹⁸ Lauritzen et al. identified 146/1,071 (14%, 95% CI, 11.6% to 15.8%) of changes to abdominal CT exam reports that were clinically significant.¹⁹ In a similar study of dual reading for thoracic CT, 91/1,023 (9%) of the report changes were clinically significant, including 3 that were critical and required immediate action and 15 that were major and required a change in treatment.²⁰ In both studies, the authors found that double reading uncovered errors with less delay and during the time when patient treatment was still able to be affected. Murphy et al. (2010), unlike the other prospective double-reading studies, evaluated blinded double reporting for patients undergoing colon CT scans. They found that, of the 24 significant findings, 7 were identified by only one of the two observers. Although this is counter-intuitive, the probability that a patient with a finding on CT examination had colon cancer was 69 percent for single reporting (11 true positives, 5 false positives) and 54.5 percent for double reporting (12 true positives, 10 false positives). For double reporting, one extra true-positive colon cancer was detected at the expense of five unnecessary colonoscopies (false positives), reducing the positive predictive value.²¹

Lian et al. (2011) compared the diagnostic error rates in a study in which CT angiograms of the head and neck were initially read by a staff neuroradiologist alone (n=144), double-read by staff and a diagnostic radiology resident (n=209), or double-read by staff and a neuroradiology fellow (n=150). Retrospectively, the CT angiograms were then blindly reviewed by two neuroradiologists to detect errors; 503 cases were included, with 26 significant discrepancies discovered in 20/503 studies (4.0%), and all errors were missed diagnoses. Ten of the 26 discrepancies were originally missed by staff alone (6.9% of studies read), 9 by staff and a resident (4.3%), and 7 by staff and a fellow (4.7%). The authors concluded that double reading with a resident or fellow reduces error.²²

1.4.3.3. Reinterpretation of Studies Conducted at Outside Institutions

Two studies examined the effect of reinterpretations of radiology studies done at outside institutions. Onwubiko and Mooney (2016) found that out of 98 reinterpreted CT scans of the abdomen and pelvis done in the context of pediatric blunt trauma, 12 significant new injuries were identified, 3 patients had their solid organ injuries upgraded, and 4 patients were downgraded to no injury. The benefit of reinterpreting scans extends beyond identifying potential diagnostic errors to limiting radiation exposure and unnecessary testing in the pediatric population.²⁴ Lindgren et al. (2014) determined the clinical impact and value of having outside abdominal imaging exams reinterpreted by subspecialized radiologists. Twenty of the 398 report comparison discrepancies (5.0%) had high clinical significance and 30 (7.5%) medium clinical significance. Over half of these discrepancies were due to overcalls, where the outside institution placed more importance on the significance of a finding than was warranted by the second review.²⁵

1.4.3.4. Unintended Consequences

1.4.4.1. Facilitators

1.4.4.2. Barriers