Abstract

Safety research in high-reliability industries, such as aviation, has clearly shown that the causes of accidents are primarily related to deficiencies in nontechnical (cognitive and social) skills, rather than a lack of technical expertise. Investigations into causes of error in anesthesia have revealed a similar pattern. The aviation industry uses behavioral marker systems to structure the training and assessment of pilot crew resource management (CRM) (nontechnical) skills. These are empirically derived taxonomies of the principal required nontechnical skills, and an observation-based rating system for assessing their component behaviors. In medicine, the enthusiasm for adopting CRM-type training needs to be underpinned by the development of a properly tested nontechnical skills framework for a given specialization. A research project involving industrial psychologists and consultant anesthetists, and funded by the National Health Service Education for Scotland, is developing and evaluating a behavioral marker system for anesthetists. It has produced and experimentally evaluated a taxonomy of anesthetists' nontechnical skills (ANTS), which can be used to guide professional syllabi, offer a method for evaluating performance, and improve the quality of feedback to trainees. This paper will provide an overview of the ANTS System and the results of usability trials conducted in the operating theatre.

Introduction

Safety research in high-reliability industries, such as aviation, has shown that accidents frequently involve deficiencies in nontechnical skills (cognitive skills such as decisionmaking or social skills such as teamworking). ¹ There is increasing awareness of a similar pattern in health care. Critical incident studies in anesthesia have found that around 80 percent of reported incidents involve human factors causes. ^{2, 3}

The aviation industry responded to this problem with the development of Crew Resource Management (CRM) training programs to provide training in nontechnical skills and promote safety behavior on the flight deck. ^{1, 4, 5} Thereafter, behavioral marker systems were developed to permit structured observation and assessment of these skills. ⁴ A behavioral marker system is an empirically derived taxonomy of the principal nontechnical skills required in a particular domain, and an observation-based rating system for assessing component behaviors. Explicit training in nontechnical skills for anesthesia has previously been limited; however, with the advent of high-fidelity patient simulators, several CRM-type programs are being developed. ^{6, 7} To be effective, such training needs to be reinforced by explicit and constructive feedback, both at the simulator and during regular training in the hospital. The Anesthetists' Nontechnical Skills (ANTS) Project in Scotland, involving industrial psychologists and consultant attending anesthesiologists, sought to develop a behavioral marker system for use in anesthesia training, both at the simulator and on the job. ⁸

A bottom-up approach was adopted for development of the ANTS System. ⁹ Three sources of data were used to identify a full set of anesthetists' nontechnical skills. First, the current state of knowledge regarding nontechnical skills and the use of behavioral marker systems in anesthesia was established, with a review of human factors research. ¹⁰ Cognitive task analysis interviews were conducted with 29 consulting anesthetists by a psychologist with a basic knowledge of anesthesia. ^{9, 11} The anesthetists were asked to recall and describe their management of a challenging case and to list the skills they considered important for good practice in anesthesia. The resulting data were analyzed utilizing a grounded theory approach to identify the nontechnical issues. ¹² Information from a survey of anesthetists' attitudes toward safety and teamwork was also taken into account. ¹³

From these data, a prototype behavioral marker system was developed by the design team according to a number of design criteria: the skills should be observable; the system should have a hierarchical structure and should be simple and usable with minimal training; and it should be complementary to the competency-based approach being adopted in medical education in the United Kingdom. The resulting prototype was then refined and condensed iteratively by making observations in the operating room, recoding a sample of the interview transcripts, and reviewing 200 anesthesia critical-incident reports to cross-check that all the nontechnical issues associated could be explained. After the production of a final prototype, examples of good and poor behaviors were selected for each element.

Lessons from the aviation industry highlight the need for such systems to be fully evaluated before use and then introduced through a well-supported process for successful integration into existing training programs. ⁴ So the next stage of the project was an experimental evaluation of the ANTS System. Video of simulated anesthesia cases showed that even with minimal training of the raters, there were sufficient levels of validity, reliability, and usability to allow further testing for usability in the real training environment. ⁸ User trials are a standard human factors research method and are extensively used to investigate usability issues. ¹⁴ In order to conduct a trial to investigate the usability issues associated with real-life training environments, it was necessary to identify the implementation requirements, such as user training, and develop user materials and guidance. This paper reports the finding of this stage of the project.

The ANTS System

The ANTS System is a behavioral marker system with a hierarchical structure starting with four main categories: task management, teamworking, situation awareness, and decisionmaking. These categories are subdivided into a total of 15 skill elements that form the main framework of the system (Table 1).

Table 1

The ANTS System: categories and elements.

Each element has a definition and sample behavioral markers (Table 2). These provide some examples of good and poor behaviors that could be associated with each skill element.

Table 2

Examples of behavioral markers for teamworking.

The system was not designed to be fully exhaustive and is limited to the principal skills that can be identified through observable behavior. Hence stress management and communication are not included in the system; stress management is difficult to observe and communication is frequently the means by which the other skills are observed.

Assessment can be made at both the element and category levels. A rating scale was developed with the system and was used during the experimental evaluation. This has four points for describing level of performance with an additional option for nonobserved skills (Table 3). The nonobserved skills option is available for use in circumstances when it is not appropriate for an element to be demonstrated.

Table 3

ANTS System rating options.

Methods

Results

Questionnaire

Thirty-one of 50 consultants (62 percent) returned questionnaires. All of the respondents said they thought the ANTS System would be useful to support training in anesthesia, although only 21 consultants reported having referred to the system since the evaluation study (9 months earlier). They reported using the ANTS System informally to support teaching, as a guide to self-reflection, and discussion of the system with colleagues and trainees.

Responses indicated that the consultants thought the system could be employed usefully at all stages of training. The consultants were asked to rate the usefulness of the system (Scale: 1 = No use; 2 = Limited use; 3 = Useful; 4 = Very useful; 5 = Extremely useful). Respondents thought the system would be most useful for structuring observations and assessments to give feedback (90 percent, n = 34), and could also be useful for structuring formal periodic assessments (65.5 percent, n = 20), for self-reflection (65.5 percent, n = 20), and structuring teaching (52 percent, n = 16) (Table 4). It was thought that the highest utility would be in simulation centers.

Table 4

Training applications for the ANTS System suggested by consultants*.

All of the respondents had previously attended a 4-hour training session as part of the evaluation study but felt that further familiarity with the system would be necessary if they were to use it comfortably and reliably in real training settings. Suggestions made for additional guidance material and training included—

• A simple handbook for use in the theater.
• Refresher training with further practice, similar to the evaluation study but also allowing more discussion of observations and ratings with colleagues, to permit calibration of standards.
• Frequent use of the system to stay current.

Asked if they could anticipate any problems with using the ANTS System in a real training setting, only one respondent answered in the affirmative. Others offered comments noting time constraints to practice and gain familiarity with the ANTS System, and difficulty to use if the consultant is heavily involved in the case or if there is an incident. Respondents thought it would not be possible to use the system when personally directing patient care, in an emergency, or when caring for a very sick patient. It would also be difficult with very inexperienced trainees, during lists requiring a rapid turnover of cases, and in other specified circumstances that were considered poor for training purposes.

Consultants (n = 21) commented on how they thought the ANTS System could be best implemented to support nontechnical skills training in anesthesia. They felt that—

• The ANTS System needs to be introduced carefully, to guide reflection or to assist consultants on a personal choice basis.
• A few consultants (for each anesthetic department) need to be trained to use the system for assessment.
• ANTS should be used mainly in the simulator environment, but could also be used to an extent for routine elective cases.

Operating room user trials

A total of 38 feedback forms were received from 17 consultant users at 5 different hospitals. One feedback form was completed on each occasion the ANTS System was used. Different participants returned different numbers of forms (mean return = 2, mode = 1, range = 1–7). A total of 24 feedback forms were received from the anesthetists in training for the 38 occasions the ANTS System was used (63 percent trainee response rate).

Overall, the system was reported to have enhanced training by both consultants (79 percent of feedback forms, n = 30) and trainees (83 percent of feedback forms, n = 20). Consultants felt use of the ANTS System was beneficial because it provided a framework for a discussion, where feedback might previously have been missed. Both trainees and consultants were able to reflect on and reinforce what makes good practice. A few trainees specifically commented that use of the system had generally raised their awareness of nontechnical skills (n = 3), that they had benefited from receiving feedback on their performance (n = 6), and that the system helped their self-reflection (n = 2).

The system was used with trainees at all training levels, with approximately equal numbers of senior house officers (trainees in first or second year), and specialist registrars (trainees in third to sixth year) (45 percent and 50 percent, respectively). The grade of trainee was not reported in 5 percent of cases.

Consultants and trainees were asked to indicate how they felt the ANTS System had been used in each session and rate this approach on a five-point scale from no use to extremely useful (Table 5). The three uses that particularly stood out for the consultants were as a debriefing/feedback framework, for postsession rating of skills, and as a language for discussion. Consultants were asked to detail whether they thought the ANTS System was useful for rating skills concurrently or after a session in order to gain insight as to what stage it was most usable for assessment of performance. Trainees were only asked if they considered the system useful for rating skills in general. As with the consultants, use of the system for debriefing and to provide a common language for discussion scored highly for the trainees. They also felt that the consultants were using the system to describe their observations, which they rated between useful and very useful, but they rated the system most useful as a tool for self-reflection.

Table 5

Uses of the ANTS System.

The ANTS System was used in its entirety 89 percent of the time. The majority of consultants (89 percent, n = 33) also reported that they used it at both category and element levels. The elements seemed to be most useful for providing specific detailed feedback about strengths and weaknesses, whereas the categories were useful for giving an overview and as a tool for more general discussions.

It was common for the system to be used with the trainee leading a case and the consultant taking a secondary role to make observations, but available to step in and take over if the clinical situation required. One consultant reported it was difficult to use the ANTS System when the consultant was required to lead the case. A different approach used with a trainee who was insufficiently experienced to take the lead in the case was to use the ANTS System as a general framework, with discussions of nontechnical skills taking place during the case as issues arose.

The consultants reported using their ANTS System observations and ratings to give feedback to the trainee—reviewing strengths and weaknesses, reinforcing good behavior, and identifying specific ways for the trainee to develop. The system was also reported as a useful way to encourage senior trainees to think about the nontechnical skills of others.

Thirteen consultants reported making notes during or immediately after the case. This was seen as important, particularly if there was a delay between the observation session and debriefing. Ratings were most commonly made after the case.

Many of the trainees reported being given the ANTS booklet in advance or receiving a presession briefing on the training aims of using the ANTS System. One trainee who had not been fully briefed reported confusion about the observation and rating process until the debriefing session. Comments from the trainees strongly suggest that they found it beneficial to receive feedback on their nontechnical skills.

With regard to difficulties in using the system, one of the consultants identified barriers to using the system on two occasions, but not on two other occasions. The problems identified by others were the need to stop using the system due to changes in the clinical situation (n = 5), having to assist directly with patient care (n = 3), and lack of time available for a debriefing (n = 2). Consultants also found it difficult not to be involved in the case themselves in order to rate the trainee (n = 2).

Trainees were also asked if they could imagine training situations when it would not be possible or appropriate to use the ANTS System. They reported that it would not be appropriate when a trainee was inexperienced in that situation (n = 2); during rapidly evolving emergencies where it might cause additional stress (n = 1); and when there were several members of the anesthesia team, making it difficult to identify the behaviors of individuals (n = 2). Two trainees gave criteria for when they thought the system should be used to avoid problems. For instance, they thought the trainee should be acting as lead anesthetist to allow the consultant to make observations independently, and that it could be used in totally new situations to maximize the chance of seeing all the elements demonstrated.

Discussion

The user trials indicate the ANTS System can be used in a real training environment to give formative feedback, providing that consultants have been trained to use the system, suitable cases can be selected, and time is available for postsession debriefing. This is acceptable to both consultants and trainees.

The most commonly employed method of using the ANTS System involved the consultant anesthetist observing the trainee during a case and making ratings once the case was over. This was most successful when the trainee could take the lead role either throughout or for a particular segment of a case, permitting the consultant to make independent observations. For this method, it is crucial that the chosen clinical situation is appropriate to the clinical knowledge and experience of the trainee. Ratings are ideally made after the case is over, when the whole performance can be analyzed. Making observation notes was found to be particularly helpful when there was a delay between the case and the feedback session. The feedback session was considered critical by both consultants and trainees.

In the majority of circumstances, the whole ANTS System was used rather than focusing on specific areas. However, the feedback naturally concentrated on those areas considered to be most in need of attention. The users found the element level was preferred over the categories level when giving detailed feedback, and also that the system was easier to use at this level. This was contrary to the finding of the evaluation study, where the trial participants perceived the category ratings easier to produce. ⁸ This was probably because the elements were rated first in the evaluation study and this assisted in making the category ratings, or it may be that behaviors are easier to observe in the real situation rather than on a video as the rater has a more complete understanding of the situation being rated. As the ANTS System is designed as an observation-based tool, it is probably best to make ratings at the element level first, as these can be linked to the observed behaviors. This will also facilitate detailed feedback. Category ratings may then be derived, if necessary.

The sample size was small, and this probably resulted from the tight time frame available for training, use of the system, and then returning of questionnaires. Those who found difficulty or were unable to use the ANTS System reported lack of opportunity as the main challenge, in that they were either not assigned trainees or the cases were not suitable. Another reason was lack of time to use the system.

The system seemed to work for all training grades, although it was used slightly more often with more experienced trainees. Perhaps this was because these trainees had the appropriate experience for a wider range of cases, and were able to lead either the whole case or a specific segment. It could be used more often as a discussion tool when a trainee does not have the relevant experience to take the lead. This could be done either as a case proceeded or during a postsession discussion. The trainee responses emphasized their need to be familiar with the ANTS System before it is used to provide feedback. Introduction of ANTS throughout the anesthesia curriculum might permit this as a framework for discussing nontechnical skills in the early stages of anesthesia education, which would progress to more observation and assessment for feedback as the trainee gained experience.

There were no problems identified with the supporting material provided for the participants (i.e., observation and rating forms and the ANTS handbook). As anticipated, in the followup questionnaires, some difficulties were encountered while using the system, but none of these were considered to be serious. It proved difficult for consultants to make observations and ratings of a trainee while participating directly in patient management. Trainees also felt that this was a situation when it would not be appropriate to use the system as an assessment tool. However, this problem is not unique to the ANTS System, but is the case with all kinds of teaching activity and would not prevent discussion of nontechnical skills issues as they arise in a case. Alternatively a trainee may be able to take the lead in some specific segments of the case, which could be an area for assessment.

The main barrier to use of the system seemed to be difficulties in securing enough time with trainees to permit a proper feedback system. Time for feedback must be scheduled to allow effective utilization of the system. Time constraints operate throughout the training process and will become more onerous with more rigid limits on both trainee and consultant timetables. ¹⁶

A major driver to initiate the ANTS Project was to develop a validated behavioral marker system to support debriefing during anesthesia simulation training, and to allow the assessment of performance to measure the impact of nontechnical skills training in the future. Results from the followup survey of consultants who participated in the evaluation study indicated that this group felt the system would be extremely useful during simulation training. The planned simulator limb of the user trials was not possible due to time constraints and changes in the pattern of specialty training in the simulator in Scotland. Despite this, the system has been used in this setting, ¹⁵ and several simulation centers in other countries are currently in communication with the project team regarding use and evaluation of the system in their own training programs.

It is important that consultant trainers are familiar enough with the ANTS System to use it easily and confidently. In addition, they require skills in giving feedback to trainees and giving concrete advice to those in whom weaknesses are identified. A critical factor in the implementation of a behavioral marker system such as the ANTS System is training of the proposed users of the system. ¹⁷ The consultants taking part in the user trials had a total of 7 hours of training on the ANTS System. The initial 4 hours of training took place during the evaluation study, but this occurred up to a year before the user trials began. In addition, design of the evaluation study had not allowed for any discussion between the consultants regarding their observations and ratings. It was clear from responses to the questionnaires that consultants felt they required further training before using the system and that time for discussion of ratings be incorporated. Clearly, the training time is not enough, and a minimum of 2 consecutive days of specialist training— which would include information on sources of rater biases and debriefing ¹⁷ —would be recommended for use of a behavioral marker system such as the ANTS System. Ideally many video examples would be taken from clinical practice; however, there are ethical and practical difficulties filming actual events. For this project, semiscripted scenarios were acted and filmed in the simulator, and this proved acceptable to the study participants.

The success of this work has now led to a similar investigation to develop a behavioral marker system for the identification of surgeons' nontechnical skills. ¹⁸

Conclusion

This practical evaluation of the ANTS System has provided information on usability of the tool in the real-world training setting, albeit with a small sample. With proper training of raters, briefing of trainees, and careful case selection, the ANTS System may offer a valuable method for facilitating structured observation and feedback of anesthesia trainees' nontechnical skills. It is recommended that the next stage of evaluation research should take the form of a larger study of both in-theatre and simulator training sessions.

Acknowledgments

This research was funded by National Health Service Education for Scotland. The views represented are those of the authors and should not be taken to represent the position or policy of the funding authority. The authors would like to thank the consultants who participated in the study and their trainees for supporting this research.

References

1.

Flin R, O'Connor P, Mearns K. Crew resource management: improving safety in high reliability industries. Team Performance Management. 2002;8:68–78.

2.

Cooper JB, Newbower RS, Kitz RJ. An analysis of major errors and equipment failures in anaesthesia management: considerations for prevention and detection. Anesthesiology. 1984;35:34–42. [PubMed: 6691595]

3.

Runicman WB, Sellen A, Webb RK. et al. The Australian incident monitoring study. Errors, incidents and accidents in anaesthetic practice. An analysis of 2,000 incident reports. Anaesth & Int Care. 1993;21:506–19. [PubMed: 8273870]

4.

O'Connor P, Höermann H-J, Flin R. et al. Developing a method for evaluating Crew Resource Management skills: a European perspective. Int J Aviat Psychol. 2002;12(3):265–88.

5.

Flin R, Martin L. Behavioural markers for Crew Resource Management: a survey of current practice. Int J Aviat Psychol. 2001;11:95–118.

6.

Howard S, Gaba D, Fish S. et al. Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med. 1992;63(9):763–70. [PubMed: 1524531]

7.

Maran N, Glavin R, Fletcher G. Training in human factors for anaesthetists in Scotland: identifying key skills and developing a training programme. Proceedings of the 7th European Form on Quality Improvement in Health Care; 2002 Mar 21–23; Edinburgh, Scotland. London: BMJ Publishing Group; 2002.

8.

Fletcher G, Flin R, McGeorge P. et al. Anaesthetists' Non-technical skills: evaluation of a behavioural marker system. Br J Anaesth. 2003;90:580–88. [PubMed: 12697584]

9.

Fletcher G, Flin R, McGeorge P. et al. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cogn Tech Work. 2004;6:165–71.

10.

Fletcher G, McGeorge P, Flin R. et al. The identification and measurement of anaesthetists' non-technical skills: a review of current literature. Br J Anaesth. 2002;88:418–29. [PubMed: 11990277]

11.

Seamster T, Redding R, Kaempf G. Applied cognitive task analysis in aviation. Aldershot, U.K.: Avebury Aviation; 1997.

12.

Strauss A, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage; 1990.

13.

Flin R, Fletcher G, McGeorge P. et al. Anaesthetists' attitudes to teamwork and safety. Anaesthesia. 2003;58:233–42. [PubMed: 12603453]

14.

Jordan P. Introduction to usability. London: Taylor and Francis; 1998.

15.

Fletcher G, Flin R, Patey R, et al. Anaesthetists' Non-Technical Skills Project Stage 3: Integration of the ANTS System into anaesthetic training. NES Project RDC/022/A Final Report; 2003. Available at: http://www​.abdn.ac.uk/iprc/ANTS.

16.

Pickersgill T. The European working time directive for doctors in training: we will need more doctors and better organization to comply with the law. BMJ. 2001;323:1266. [PMC free article: PMC1121739] [PubMed: 11731377]

17.

Klampfer B, Flin R, Helmreich RL, et al. Enhancing performance in high risk environments: recommendations for the use of behavioural markers. Ladenburg: Daimler-Benz Shiftung; 2001. Available at: http://www​.psyc.abdn.ac.uk/serv02.

18.

Flin R, Yule S, Paterson-Brown S, et al. Rating observations of surgeons' non-technical skills. Anesth & Analg 2004;98(5 suppl): May.