U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Henriksen K, Battles JB, Marks ES, et al., editors. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005 Feb.

Cover of Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology)

Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology).

Show details

Development and Implementation of The University of Texas Close Call Reporting System

, , , , and .

Author Information and Affiliations

Abstract

This report describes the development and implementation of The University of Texas Close Call Reporting System, an anonymous tool used to collect information about potential errors that do not reach the patient. The system was originally developed through experimentation, using a fund of knowledge collected in the aviation industry. A multidisciplinary health care team adapted this approach to develop a close call categorization scheme based on human-factor principles. This system, which is available in Web-based and paper formats, allows reporters to submit close calls in a risk-free manner and to enable the collection of rich information about the etiology of close calls. This information is used to identify areas of vulnerability and to develop interventions that can prevent problems from recurring. The development and implementation processes presented provide a comprehensive framework that can be used for future deployments of similar patient safety systems.

Introduction

The Institute of Medicine (IOM) reports To Err Is Human: Building a Safer Health System 1 and Crossing the Quality Chasm 2 identified several ways that health care professionals might increase patient safety. One method discussed in the IOM reports and other recent publications is a reporting system that focuses on errors and close calls. 3 – 7 Asking health care providers to report errors and close calls is an excellent way to discover events not revealed by other methods and to capture important information about the underlying causes of these events. 1, 8 – 10 Furthermore, reporting systems are expected to identify opportunities for process improvement and to facilitate the development of innovative interventions that can prevent the recurrence of these events. 11, 12 Given the importance of reporting systems and their possible benefits, The University of Texas Close Call Reporting System (UTCCRS) was developed.

As the name implies, the focus of the UTCCRS is to collect information about close calls, which are defined as potential errors that are caught and prevented before reaching the patient or causing harm to the patient. Close calls were chosen as the focal point of the reporting system for several reasons. First, given that close calls are believed to have underlying causes similar to those for errors, 11, 12 a reporting system that focuses on close calls should allow researchers to understand the etiology of both close calls and errors. Second, because close calls are theorized to occur much more frequently than errors, 11 – 13 it is expected that many areas in need of process improvement would be identified. Such identification can then be used to develop and test the efficacy of interventions in preventing the recurrence of similar close calls and errors. 13 Third, while many health care institutions and government agencies have error reporting systems, few close call reporting systems exist. 4 One reason, therefore, to develop a system focused exclusively on close calls is to determine the utility such a system might have for patient safety. Additionally, focusing on close calls represents a paradigm shift for the health care industry in which health care is moved from a position of reactivity to proactivity. In sum, a close call reporting system is expected to provide health care researchers and professionals with information about the etiology of close calls, areas for improvement, and potential interventions for future events, thereby increasing patient safety. The purpose of this report is to detail the replicable process used to develop and implement the UTCCRS.

Phase one: system development

The initial phase of the development of the UTCCRS consisted of two complementary activities: (1) forming an advisory board that would provide input and guidance about the development of the UTCCRS; and (2) reviewing the research literature on reporting systems. A multidisciplinary advisory board was appointed, and included health care administrators, nurses, pharmacists, and physicians. Given that the UTCCRS project is one of five projects conducted through the University of Texas Center of Excellence for Patient Safety Research and Practice (whose principal investigator is Eric J. Thomas of The University of Texas-Houston Medical School), funded by the Agency for Healthcare Research and Quality (AHRQ), experts from the Center also participated on the advisory board. These experts provided valuable information about translation of best practices from aviation to health care, human factors, organizational learning, and safety attitudes throughout the development process.

At the initial advisory board meeting, the members were informed about the concept of a close call reporting system and reasons that such a system would be valuable in a health care setting. Additionally, the board was provided with an overview of the systems perspective discussed in To Err Is Human. 1 The advisory board was asked to provide expert opinion about issues identified from the literature 13 that are important to consider when developing a reporting system, including: (1) a definition of a close call; (2) potential barriers that might be encountered with a close call reporting system (i.e., whether reporting should be anonymous or confidential); (3) possible modalities of the system (i.e., paper, phone, electronic); (4) feedback mechanisms to reporters; and (5) the format and content of the system.

Definition of a close call

Given that the UTCCRS focuses on close calls, it was important that the advisory board agree on a close call definition. Surprisingly, many members disagreed about what constituted a close call versus an error, with definitions varying, based on the slippery slope of “no harm.” The advisory board adopted Bagian and Gosbee's definition of a close call, an “event or situation that could have resulted in an accident, injury, or illness, but did not, either by chance or through timely intervention.” 3 The adopted close call definition was also favored because it represented a conservative conceptualization of close calls, was developed by recognized leaders in patient safety initiatives, and was expected to make reporters feel comfortable reporting potential errors as opposed to actual errors.

Potential barriers to reporting

The advisory board discussed reporter anonymity—where the reporter cannot be linked with a report, versus confidentiality—where the reporter-report link can be established. Consistent with research literature on this topic, 1, 3, 4, 14, 15 the advisory board was divided about whether to develop an anonymous or a confidential system. The proponents for a confidential system argued that it was essential to followup with reporters and provide them with feedback. Advocates of the anonymous system noted that such systems would be perceived as safer by reporters and would reduce the fear of punitive measures. The advisory board identified a third option—a hybrid system that was confidential until the report was completed and reviewed, at which time the report would be de-identified and entered into the system database. While advisory board members noted that the decision of which system to adopt should be based on which system would maximize reporting of close calls, there was no reliable research found on the reporting rates of the three options. The members decided unanimously to adopt an anonymous system to create the safest environment for reporters to provide information about close calls.

System modalities

The advisory board was also asked to determine possible modalities for the UTCCRS. Anecdotal comments from the advisory board led to the belief that different professions use different modalities. For example, comments revealed that physicians and pharmacists might favor electronic and telephone reporting systems, while nurses might favor paper forms. Advantages to a Web-based system included the ability to access it from any computer and that a short amount of time is needed to complete such a report. A telephone system was viewed as a plausible solution because it could be designed similarly to transcription services, which are already used by health care providers. Paper forms had the advantage of being easy to store in health care institutions that lack a large number of computer resources and are the most familiar report format for many providers. The advisory board members decided that the feasibility of all modalities should be examined. After examining the specifications needed to get all three modalities working, it was decided to eliminate the telephone modality because of concerns about (1) ease of use; (2) time to report; and (3) assurance of anonymity. It was decided to adopt both the electronic and paper reporting modalities because both met these specifications. The electronic form was implemented on a secure Web site, and the paper form consists of a double-sided, one-page report.

Feedback to reporters

Researchers mentioned feedback as the key attribute in obtaining multiple reports from providers. 3 – 5 Not surprisingly, the advisory board also mentioned the necessity of feedback to close the information exchange loop with the reporter and to ensure system success. However, given the anonymous nature of the system, feedback seemed problematic. To address this issue, a method of feedback was developed for both the individual (i.e., reporter) and aggregate level. At the individual level, after a reporter enters a report electronically, he or she is given a random sequence of numbers as a specific identifier for that report. The reporter can then visit the UTCCRS Web site and track his or her report by entering the random sequence of numbers. The tracking feature allows researchers to provide specific feedback to the reporter about next steps that would result from the close call report, such as any best practices that may be developed to address the close call. The feedback mechanism for the paper form works similarly, with preprinted random numbers on the paper form that can be tracked via the UTCCRS Web site. At the aggregate level, feedback sessions for participating institutions were discussed, with members of the UTCCRS team providing frontline providers with information about themes and trends from an institution's close call data.

Format and content

An examination of the literature on elements common to existing reporting systems revealed a lack of consensus concerning whether short responses, longer or narrative responses, or check boxes should be used to collect information about a situation. Researchers in favor of check boxes noted that check boxes might speed up the time it takes to complete a report, which was mentioned as a possible corollary to increased reporting. Researchers favoring narration argued that this response format produced the richest type of data. Interestingly, no research was found that demonstrated a significant difference in the “richness” of the data; rather, the noted differences were anecdotal. Given the lack of consensus in the literature concerning response format, the advisory board recommended a hybrid approach using both check boxes and narrative responses.

With respect to system content, the advisory board members noted that the primary areas of interest should include time and location of the close call; reporter demographics; description of the close call; type(s) of close call; and reasons for close call occurrence. The advisory board recommended the formation of a multidisciplinary work team to determine specific questions for these areas of interest. A work team was formed, consisting of practicing health care providers, that met weekly for 1 year.

Phase two: consensus about content

The work team conducted an indepth examination of reporting systems—including some site visits in high-risk industries like aviation, petrochemicals, the military, and health care—to better inform its decisions about system content. This review resulted in the following summaries:

  • Aviation Safety Reporting System (ASRS). The ASRS is administered by the National Aeronautics and Space Administration and offers a level of anonymity (after the report is filed) and confidentiality. Because the ASRS is designed for reporting all nonaccidents (close calls), it provides an opportunity for more prospective review and preventive intervention. The ASRS receives upwards of 30,000 reports per year. 16
  • Aviation Safety Action Partnership (ASAP). This new system permits pilots to report incidents to their own companies with the same limited immunity provided by the ASRS national program. The ASAP form asks the reporter to categorize the incident by human-factors categories, as opposed to relying only on an event narrative.
  • Prevention and Recovery Information System for Monitoring and Analysis. This system collects data on all accidents and close calls within the petrochemical and steel industries. Anonymity and confidentiality are maintained.
  • Nuclear Regulatory Commission Allegations System. This reporting system is maintained by the Nuclear Regulatory Commission and monitors “all safety concerns” within the nuclear industry. Anonymity is not guaranteed, but reporter immunity is provided.
  • Patient Safety Reporting System (PSRS). This reporting system, used by the Veterans Health Administration, emphasizes voluntary and confidential reports of nonintentionally unsafe close calls and events within a nonpunitive environment.

Additionally, to aid in the development of the UTCCRS, the work team asked for an analysis of the report format and results generated from the current incident report system at The University of Texas M. D. Anderson Cancer Center. The analysis included a Pareto chart of categories, subcategories, and contributing factors from M. D. Anderson's incident reports, as well as the background and demographic questions included in the reports. Such analyses informed decisions about which types of categories (i.e., medication, equipment problems, etc.) and demographic questions to include in the UTCCRS; the work team's decisions about background and demographic questions and close call categories are presented in Table 1. The work team decided to include free-text narrative fields, based upon the ASAP and PSRS models, asking the following questions: (1) describe the event; and (2) what could be done to keep it from recurring?

Table 1. Primary information categories.

Table 1

Primary information categories.

The work team struggled to identify which contributing factors to use to recognize the etiology of close calls with respect to human and systems factors. The lack of consensus about contributing factors played a part in the richest experience of the present study: The team used brainstorming and affinity problem-solving methods to achieve consensus. The initial brainstorming session resulted in 8 broad contributing factors, while the second session—which occurred 1 month after the first session—resulted in 18 specific contributing factors. However, there was a great deal of confusion about the specific meaning of the factors.

Given the confusion, several sentences were written for each factor that provided detailed information, were easy to understand from the reporter's perspective, and were less open to multiple interpretations because of their specificity. The contributing factor names were removed; the sentences were written on sticky notes and put on the walls of a conference room, and the work team members placed the sentences from the contributing factors into natural groups (i.e., affinity). The affinity exercise resulted in 17 groupings, with each having between 4 and 9 sentences. Names were generated for each grouping (e.g., contributing factor); Table 2 details the change in contributing factors across the three iterations. An Appendix (available at the Web site https://www.utccrs.org/ ccrs/Publications.html) contains contributing factors and associated drill-down sentences. Once the work team reached consensus about reporting system content, a technical development team was asked to translate the work team's vision of the UTCCRS into a fully functioning reporting system.

Table 2. Contributing factors identified as the result of the brainstorming and affinity exercises.

Table 2

Contributing factors identified as the result of the brainstorming and affinity exercises.

Phase three: UTCCRS prototype

The ASAP model from aviation was used as a template for the UTCCRS prototype, which was developed using Microsoft™ Access. While developing the prototype, the technical development team commented that the system appeared to be visually cluttered because of the number of background and demographic questions, categories, contributing factors, sentences associated with the contributing factors, and narratives. This issue was presented to the work team, who decided that the system should employ a series of drop-down menus that would limit the amount of information that the reporter had to view at a given time. While drop-down menus made the screen much less cluttered, the approach was not user-friendly because one had to continually scroll up and down the drop-down menus to select all of the appropriate categories, contributing factors, and contributing factor sentences. The work team was consulted again and asked to propose an alternative layout for the system. The work team agreed that the reporting system should be divided into five main screens:

1)

Background and demographic questions

2)

Narratives

3)

Close call categories

4)

Contributing factors

5)

Contributing factors sentences based on conditional-logic heuristics

The work team then discussed mechanisms by which to ensure that only close calls were reported in the system. The suggestion that was implemented was to include a trap question at the beginning of each report to ensure that only close calls, based on the definition adopted by the advisory board, were reported. The question asks “Was the action caught before it was carried out and could reach the patient?” A “yes” response to this question allows the reporter to continue in the system; a “no” response exits the reporter out of the report system and informs them to use an incident report. Additionally, the narrative section contains specific instructions to reporters that they should omit any information that might identify patients or providers, either by name or by identification number. To further ensure de-identified information, the system is designed so that reports go initially to a temporary database where a clinical analyst reviews the data and removes any identifiers, and then the data are sent to the final database.

The technical development team then decided it was necessary to field test the Microsoft Access prototype with frontline providers who had not participated on the work team. Nurse, pharmacist, and physician volunteers were asked to test the UTCCRS prototype and enter data from hypothetical scenarios. While entering data, these providers were asked for their ideas and concerns about system design from a user perspective. Human-factors analysis using standard methodology for computer systems was also conducted. All of these findings led to the development of a beta model using Web-based Java™ and Oracle applications. This set of applications allowed for the creation of several enhancements that responded to user observations and the human-factors analysis. Key enhancements included—

  • a frequently-asked-questions (FAQ) section in the electronic version that provides information about aspects of the reporting system;
  • a help section in the electronic version that contains definitions for the primary terms relevant to the reporting system;
  • report-editing capability, where the final screen of the report displays all of the reporter's responses and contains hyperlinks to each section of the report that the reporter can use to change his or her responses; and
  • enhanced security options in the electronic version to eliminate the possibility of hackers overloading the system with automated report writers.

At the end of the development of the beta model, additional volunteers were approached and asked to enter scenario data. Review of the reporting system was conducted again using standard human-factors methodology for computer systems. Overall, the response from users was overwhelmingly positive with respect to ease of use, design of the system, and intuitiveness of the system. The work team and advisory board reviewed the final version of the reporting system and provided their approval. Once the final electronic version of the UTCCRS was approved, it was deployed to a secure Web site, https://www.utccrs.org/. The paper form was then developed to reflect the same questions contained in the electronic version of the reporting system. Final versions of the reporting system allowed for system implementation. The following sections detail the UTCCRS implementation process.

Phase four: implementation

Preimplementation

Administrators from a teaching health care institution in Texas discussed the potential usefulness of a close call reporting system during a meeting in June 2002. This institution is in a moderately sized city, and supports the surrounding rural areas. During the next 4 months, the UTCCRS research team collaborated with key personnel from the institution to develop a research protocol for the reporting system. A followup meeting occurred in October 2002, which resulted in a finalized research protocol and led to the development of a project plan for deployment of the reporting system. This plan focused on discussing the reporting system with frontline staff and managers, and identifying and addressing barriers that might hinder implementation of the project.

Presentations were made to a representative sample of frontline staff and managers about the concept and purpose of the UTCCRS, expected systems interventions that would be developed based on the reports, and the expected impact on their overall patient safety efforts. The managers identified several concerns about the reporting system: (1) fictitious reports, (2) reporting of errors instead of close calls, and (3) insignificant reports. Frontline staff mentioned concerns about (1) anonymity, (2) punishment by a supervisor if the report was recognized, (3) whether the Web-based system could be tampered with, and (4) when the system should be used. All issues mentioned by managers and staff were discussed and clarified to create a nonthreatening environment prior to deployment of the UTCCRS.

System launch

The reporting system was launched at the teaching health care institution in March 2003. A physician from the participating institution provided an overview of the UTCCRS at a meeting for physician staff, while the UTCCRS team provided educational sessions for other health care professionals at all shifts over 3 days. Two members of the UTCCRS team—a clinical nurse-specialist and an organizational psychologist—conducted the educational sessions. In addition to using standardized materials to explain how and when to use the reporting system, the educators also provided a context for the importance of a close call system in today's health care system, including references to the Institute of Medicine reports, lessons learned from the aviation industry, and the similarity of underlying causes for close calls and errors. Additionally, because the power of storytelling has been demonstrated as an effective way to educate staff about new issues, 17 anecdotal stories reflecting nurses' expertise in catching close calls were shared with staff during the educational sessions to draw parallels between concepts from the UTCCRS and examples from their clinical practice. Providers were informed that they could report close calls via a paper form with accompanying postage-paid envelopes or electronically from any computer connected to the Internet (by accessing the Web site https://www.utccrs.org/). Given the UTCCRS project's goal of implementing the reporting system in multiple institutions and the need for reporter anonymity, institution codes were employed. Institution codes are a string of letters and numbers used by all reporters at an institution. The codes maintain reporter anonymity while providing the UTCCRS team with information that allows them to give feedback to the correct institution.

Action planning

Electronically submitted reports are communicated to the UTCCRS team in real-time via e-mail. The clinical analyst of the UTCCRS team screens all reports to ensure that they convey an event that is consistent with the definition of a close call. Reports inconsistent with the definition are communicated back immediately to a quality liaison from the participating institution. The clinical analyst communicates any urgent close calls back to the institution; “urgent” is defined as a situation where immediate action is needed to prevent harm to a patient in some part of the institution. For example, two drugs stored in the wrong bins of a medication-dispensing machine might be reported through the system. Because incorrect storage in one machine might mean incorrect storage in another machine at an institution, the clinical analyst from the UTCCRS team would inform the quality improvement/quality assessment (QI/QA) liaison of that institution of the problem. It is possible that the liaison of that institution, upon examining the other medication-dispensing machines in the institution, might discover the same problem in multiple machines. The clinical analyst also de-identifies any reports that inadvertently contain provider or patient identifiers, prior to entering the information into the final database, and prepares feedback reports for the institutional liaison of the participating institution.

Feedback

During the first year of UTCCRS implementation at the participating institution, feedback sessions were developed and offered on a quarterly basis. The feedback sessions consisted of 10–15 minute presentations highlighting the content of the reports and the action plans developed by the institution to address some of the issues mentioned in the content. Examples of these action plans are failure mode and effects analysis; newsletters highlighting medication tips; education sessions from the pharmacy about different types of drugs; and online patient safety education.

Preliminary findings about UTCCRS and conclusion

To assess the utility of the UTCCRS in terms of ease of use and impact on patient safety efforts, an optional anonymous survey was added to the electronic version of the reporting system. Nineteen providers have completed the survey as of March 2004, and their responses have been overwhelmingly positive. The survey items were based on a Likert scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, and 5 = Strongly Agree. The items and their respective mean (M) and standard deviation (SD) are as follows:

  • I believe the results from the UTCCRS will be used to develop interventions that will make health care safer (M = 4.21; SD = 0.79)
  • I am satisfied with the UTCCRS (M = 3.8; SD = 0.89)
  • I plan to recommend the UTCCRS to my colleagues (M = 4.33; SD = 0.66)
  • The UTCCRS asks confusing questions (M = 2.0; SD = 0.86)
  • I believe the hospital will be a safer place by having the UTCCRS (M = 4.0; SD = 0.79)
  • The UTCCRS is easy to use (M = 4.25; SD = 0.97)
  • I plan to use the UTCCRS again to report a close call (M = 4.4; SD = 0.68)

In sum, the UTCCRS is a proactive patient safety tool that allows institutions to learn about the etiology of close calls. Reporters' feedback about the system has demonstrated that the system is easy to use and is perceived as a positive influence on patient safety efforts. To date, the UTCCRS has been implemented in six institutions using the implementation and maintenance plan described previously. Currently, the project team is formalizing plans to share themes and lessons learned across institutions so that interorganizational learning and a simultaneous increase in patient safety may occur.

Acknowledgments

This research was supported by a grant from the Agency for Healthcare Research and Quality (Grant No. 1PO1 HS1154401).

The authors wish to thank the following contributors for their input and guidance: Pam Bennett, Department of Veterans Affairs; Michelle Harper and Robert Helmreich, The University of Texas Human Factors Research Project; Captain Bruce Tesmer, Continental Airlines; and Eric Thomas, The University of Texas-Houston Medical School. Additionally, the authors thank the UTCCRS Advisory Board and Work Team for their efforts during this project.

References

1.
Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer healthcare system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2000.
2.
Kohn LT, Corrigan JM, Donaldson MS, editors. Crossing the quality chasm: a new health system for the 21st century. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2001.
3.
Bagian JP, Gosbee JW. Developing a culture of patient safety at the VA. Ambulatory Outreach, 2000 Spring; 25–9. [PubMed: 11067444]
4.
Barach P, Small SD. Reporting and preventing medical mishaps: lessons from nonmedical near miss reporting systems. BMJ. 2000;320:759–63. [PMC free article: PMC1117768] [PubMed: 10720361]
5.
Golden MS. An incident reporting system: documented at the point of service. J Healthc Risk Manag. 1998 Spring;2:18–26. [PubMed: 10185074]
6.
Kivlahan C, Sangster W, Nelson K. et al. Developing a comprehensive electronic adverse event reporting system in an academic health center. J Qual Improv. 2002;28:583–94. [PubMed: 12425253]
7.
Pronovost P, Wu AW, Dorman T. et al. Building safety into ICU care. J Crit Care. 2002;17(2):78–85. [PubMed: 12096370]
8.
Brennan TA, Lee TH, O'Neil AC. et al. Integrating providers into quality improvement: a pilot project at one hospital. Qual Manag Healthc. 1992;1:29–35. [PubMed: 10131644]
9.
Petersen LA, Brennan TA, O'Neil AC. et al. Does house staff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866–72. [PubMed: 7978700]
10.
Petersen LA, Orav EJ, Teich JM. et al. Using a computerized sign-out to improve continuity of inpatient care and prevent adverse events. Joint Commission Journal on Quality Improvement. 1998;24(2):77–87. [PubMed: 9547682]
11.
Reason J. Human error. Cambridge, UK: Cambridge University Press; 1990.
12.
Reason J. Managing the risk of organizational accidents. Aldershot, UK: Ashagate; 1997.
13.
Anderson DJ, Webster CS. A systems approach to the reduction of medication error on the hospital ward. Journal of Advanced Nursing. 2001;35:34–41. [PubMed: 11442680]
14.
O'Leary M, Chappell SM. Confidential incident reporting systems create vital awareness of safety problems. ICAO J. 1996;51:11–13. [PubMed: 11541832]
15.
Runciman WB, Sellen A, Webb RK. et al. Errors, incidents and accidents in anaesthetic practice. Anaesth Intens Care. 1993;21:506–19. [PubMed: 8273870]
16.
Johnson CW, McElroy P. Supporting the analysis of human error in national and international incident reporting schemes. In: Cacciabue PC, editor. Proceedings of the 19th European Annual Conference on Human Decision Making and Manual Control. Ispra, Italy; 2000. pp.137–44. (Available at: http://www​.dcs.gla.ac​.uk/~johnson/papers/Inquery)
17.
Benner P. From novice to expert: excellence and power in clinical nursing practice. Menlo Park, CA: Addison-Wesley Publishing Company; 1984.

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this page (186K)

In this Page

Other titles in this collection

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to PubMed

Similar articles in PubMed

See reviews...See all...

Recent Activity

ClearTurn OffTurn On

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...