4Discussion

Principal Findings

The extensive literature search identified 21 articles that described and evaluated an educational intervention designed to train hospital staff to respond to an MCI. The majority of these studies addressed the use of hospital disaster drills as a training tool (key question 1).

The studies represent a heterogeneous body of literature, ranging from descriptions of local drills to sophisticated telecommunications exercises. Studies also varied in terms of targeted staff, learning objectives, identified outcomes, and evaluation methods. Because of the wide range of foci for the studies, it is difficult to make definitive recommendations on the most effective approaches for training clinicians to respond to an MCI. However, some potentially valuable points can be identified in the literature.

Lessons Learned Based on Outcome Categories

Limitations

This evidence report has a number of limitations, of which the most obvious is the small number of studies that were directed to the training of hospital staff in how to respond to MCIs. In addition, the search was limited to published English language articles. There may be classified, unpublished material or studies in press that were not included in this report. An example would be material from the U.S. Department of Defense, which undoubtedly has experience at testing different scenarios, but these materials are not available in the published literature. There may be aspects of military disaster drills that have potential applications for hospitals. Another example is unpublished results of drills and exercises associated with JCAHO requirements taking place at state and local levels. It is not known if this unpublished material includes evaluation data.

Another major limitation relates to the fact that different types of disasters raise different issues for training of hospital staff. For example, the issues differ for drills that simulate a transportation incident or fire and those that simulate a biological incident. The latter most likely would evolve over an extended time period while the former would introduce a sudden influx of cases to hospitals. Since nearly all of the published studies focused only on training in how to respond to conventional types of disasters, little direct evidence exists on the effectiveness of training hospital staff in how to respond to a biological MCI.

Although many experts believe that tabletop exercises have an important role to play in disaster preparedness,^37–40 the literature search identified only one study that evaluated use of a tabletop exercise for training of hospital staff in disaster preparedness.³⁶ One other study reported on the use of a tabletop exercise to provide disaster training to emergency medical technicians (EMTs).⁴⁰ However, this study did not meet our eligibility criteria because it did not involve hospital staff. In this study, Chi assessed the attitudes of EMTs toward tabletop exercises. Survey results showed that EMTs believed that tabletops performed better than field exercises in linking the results of disaster exercises to appropriate changes in terms of training, equipment, and supplies. Other tabletop exercises have been described in the literature,^{38, 39} but none of these reported any evaluation data. Without evaluation data, one cannot draw conclusions about the effectiveness of such exercises. Although numerous tabletop exercises have been conducted as a less expensive alternative to operationalizing drill training, most of the identified studies on tabletop exercises were not focused on hospital-based activity, and among those that were, no results, i.e., no data, were given.

The quality and methodological limitations of the studies make it difficult to judge external validity of results. Furthermore, marked differences in educational interventions, objectives, targeted audience, and evaluation methods present challenges in drawing generalized conclusions relevant to bioterrorism preparedness. Another specific limitation is that the search identified only one evaluation of a tabletop exercise, one of a computer simulation, one of video training, and one of teleconferencing. Although each of these educational techniques may have distinct advantages, the evidence is insufficient to draw definitive conclusions about their effectiveness.

In addition, the financial burden of the educational interventions generally was not reported. Full-scale disaster drills are expensive.²⁷ The large-scale three-day drill described by Inglesby et al. cost $3 million (U.S.). None of the other studies identified cost figures, thereby leaving a gap in this important aspect of hospital disaster preparedness. Finally, very few studies identified the organizing or sponsoring entity (e.g., federal, state, or local agency, or hospital) for the drill or exercise, thereby precluding any conclusions about who most effectively plans and conducts drills.

Future Research

Part of the challenge in reviewing the existing literature regarding training of hospital staff to respond to an MCI arose from the numerous formats of studies and differences in evaluation methods. Creating a template for future training reports (e.g., Utstein-style guidelines)⁴¹ may facilitate the accessibility, synthesis, and interpretation of collected data. Authors of future reports should consider the merits of adopting a common nomenclature and explore establishing the incident command system among hospitals as a standard.

One major issue is the cost of conducting drills and the need for a steady stream of funding to support these activities. Although governmental funding has increased recently, many hospitals are short of flexible funding and are unable to assign high priority to disaster preparedness. Given the evidence on the potential value of drills, the recent international events that indicate an increasing likelihood of future MCIs, and the heightened focus of the government in providing funding, it has become a priority to explore the uses of drills. Evidence is needed. One approach might be to provide funding for hospital exercises that are designed to overcome the limitations identified here.

The purposes of drills are important and in general underexamined. Drills may have many different purposes and it is imperative to conduct different types of exercises to test the different operational elements involved. Valid purposes include testing communications, triage planning, evacuation or decontamination procedures, and focusing on improving familiarity with emergency protocols. Related issues in drilling include the necessity to be efficient and to incorporate continuous training to meet the needs related to turnover in the hospital workforce. To develop a drill, major preparedness issues should be identified and then tested in different types of drills; however, hospitals should remain open to learning from the unexpected that occurs during the course of a drill as well. Overall, it is important to follow the principle of learning from the experience without judging the drill as a success or a failure.

The current evidence is not definitive on the effectiveness of hospital disaster drills in training staff to respond to an MCI. Although hospital disaster drills arguably may provide the most realistic training, they also represent a resource-intensive training format for MCI preparation. To date, no evidence supports the cost effectiveness of any particular type of training intervention. Future studies addressing the costs of educational interventions will facilitate recommendations regarding training strategies. For example, the strength of a video is that it is a relatively inexpensive way to standardize training for a large group (hospital employees or others) who have different schedules and operate remotely from each other. A good tabletop exercise allows observers to see the action develop, to gain increased awareness, to build teamwork, and to test strategic scenarios. Tabletops are economical and more efficient for some purposes. Given these attributes of these different types of exercises, a logical progression to familiarize employees with a hospital disaster drill plan may be to use videos in a group setting, then demonstrate key points with a tabletop exercise, and later graduate to a fully operationalized partial or complete drill.

Another major issue that needs to be addressed is the lack of evaluation of completed drills, and an equally important issue is the dearth of published reports from individual hospital- or health department-supported drills. Very few reports of hospital disaster drills have been ublished or made available in electronic databases of relevant material. Therefore, hospitals and other agencies are denied the benefit of others' experience. In the future, agencies or institutions funding drills may chose to prioritize both evaluation and more rigorous written reports post-drill. Federal agencies might direct grantees to document their findings and submit them for publication. As drills of some nature are now mandated by JCAHO, hospitals should be able to generate an increased number of reports about what does and does not work well. JCAHO may also want to encourage hospitals to meet the drill requirements through other than standard (i.e., non-disaster-related) training exercises.

Translating the reports of drills into future activity may help to promote orderly development of capability in the field. Disaster drills might be designed to test specific elements of response as identified in this report, including incident command system, communications, triage, patient flow, tracking, security, materials and resources, and decontamination.

More attention should be given to evaluating the effectiveness of relevant training programs in a scientifically rigorous manner. The weak study designs led to insufficient strength of the evidence to draw firm conclusions. This is typical of the present literature in disaster medicine and points to a need for better_designed studies.

This synthesis of the existing evidence on the implementation and evaluation of hospital disaster drills, computer simulations, and tabletop or other exercises may help to establish criteria for assessing the effectiveness of future training exercises. Because the current evidence on tools or methods used to evaluate effectiveness of training hospital staff to respond to an MCI is insufficient, development of a modular evaluation tool to address the effectiveness of different educational interventions will be of significant importance. These modules could follow the elements of response identified above (e.g., incident command system, triage, treatment, communication, and security).

Finally, the published evidence includes very little information that directly pertains to the training of hospital staff in how to respond to a biological MCI. Although bioterrorism is a current federal priority, only one study described the response to a mock bioterrorist attack.²⁷ The majority of studies focused on more common disasters (e.g., a fire/explosion or transportation accident).^{18–20, 22, 23, 25, 26, 30, 32, 33, 35, 36} It seems reasonable to postulate that preparedness for a conventional MCI would enhance readiness for unconventional MCIs. For example, an effective response to a bioterrorist event will depend on the general training and preparedness necessary for other MCIs, including training on communications, triage, and treatment during an MCI. However, a biological event would differ from a conventional MCI in important ways, such as evolving presentation of cases over days or weeks with a potentially high casualty toll, coupled with a large number of concerned and potentially exposed victims. Future training should therefore also address biological as well as chemical, nuclear, or radiation events.

Disaster training activities at the local, state, and national level could improve our knowledge pertaining to drill training, but they have not yet been reported consistently in the peer-reviewed literature. This lack of information suggests a need for creating improved ways of sharing such training experiences. Future disaster preparedness would be facilitated by a systematic method for collecting this information and making it readily available for review and synthesis.