Recruitment of Experts

The JHU EPC team identified 12 experts to provide input at key points during the project (see Appendix A). These included two representatives of relevant professional organizations, two experts representing government agencies, and eight experts from academic settings. The experts participated in the task of refining the key questions (see Identifying the Specific Questions, below), and they also reviewed the draft report (see Peer Review Process, below).

Target Population

The target population addressed in this evidence report consisted of hospital staff who participated in an educational intervention related to MCI response. For the purpose of this report, hospital staff included all clinical, non-clinical, and administrative staff.

Identifying the Specific Questions

The Health Resources and Services Administration (HRSA) developed the initial list of questions to be addressed. The EPC team refined the original questions through analysis of preliminary literature searches and input from the experts.

Analytic Framework

The JHU EPC team developed an analytic framework (see Figure 1) to depict the central role the hospital will play in responding to an MCI. The framework illustrates the complex nature of such an event and the elements of hospital disaster response that have been identified as important. This complexity underscores the need for developing and testing hospital disaster plans. If an MCI occurs, the hospital will be at the center of all operations regarding victim care, yet it must be in contact with the local emergency services, other hospitals, and city, state, and federal agencies. Coordination of the entire incident will in many cases be through the public health system or the government, and so communications is a key area. Materials, equipment and supplies, and extra personnel will be drawn from outside the hospital, as well as from within. News media, family, and other area residents will impose an additional outside burden on hospital operations that must be managed. Inside the hospital, an incident command system will be needed for communication with all clinical care areas and hospital departments such as security and central supply.¹⁰ The incident command system will address the need for and implementation of all disaster response activities. During the period of the MCI, the hospital will attempt to continue to deliver needed services as required (not depicted here).

Figure

Figure 1. Analytic framework depicting the hospital's role in responding to an MCI.

Literature Search Methods

The literature search consisted of several steps, including identifying sources, formulating a search strategy for each source, and executing and documenting each search.

Sources

Several literature sources were used to identify all studies potentially relevant to the key questions. Both electronic database searching and hand searching were completed. Six electronic databases were searched. The databases included PubMed^®, the Cochrane Central Register of Controlled Trials (CENTRAL), the Excerpta Medica database (EMBASE), the Educational Research Information Clearinghouse (ERIC), the specialized register of the Effective Practice and Organization of Care Cochrane Review Group (EPOC), and the Research and Development Resource Base in Continuing Medical Education (RDRB/CME). The electronic searches were conducted in December 2002, with an updated search of PubMed in February 2003, and no restrictions based on publication date were used.

EPC team members also hand searched the literature to ensure comprehensiveness. Team members reviewed the reference lists of relevant reviews, reference papers, and the eligible articles. Team members also hand searched the most recent issues of journals (through January 2003) frequently identified by the electronic search and/or identified as high priority by the team (see Appendix B).

Search Terms and Strategies

The search strategies were designed to maximize sensitivity and were developed in consultation with team members. Key articles were identified from the previous EPC project.⁸ Using these key articles determined to be eligible for review, search strategies were developed and refined in an iterative process. A strategy was first developed for PubMed^® and modified for use in the other electronic databases. The strategy used text words and controlled vocabulary words, such as mass casualty, disaster, disaster planning, and drill. All electronic database search strategies are included in Appendix C.

Organization and Tracking of Literature Search

The results of the searches were downloaded from electronic sources whenever possible or, if necessary, manually entered into a ProCite^® database (ProCite, ISI ResearchSoft, Berkeley, CA). The ProCite^® database was used to store citations and track search strategies and sources. The software was also used to track the abstract review process.

Abstract Review Process

Two members of the study team independently reviewed each abstract identified by the search. For each team of reviewers, one reviewer had training in emergency medicine and one had training in epidemiology and research methods. Team members applied the following criteria to exclude articles from further consideration:

1. not written in English;
2. did not include human data;
3. no original data;
4. meeting abstract (no full article for review);
5. did not include hospital staff;
6. did not include response to an MCI or a disaster;
7. did not include training or education;
8. no evaluation of the training or education; or
9. did not apply to any of the key questions

A copy of the abstract review form is included in Appendix D. Disagreements about the eligibility of an article were adjudicated by consensus.

Qualitative and Quantitative Data Abstraction

The EPC team developed and pilot tested two article review forms. The quality assessment form and the content abstraction form are included in Appendix D.

The quality assessment form asked questions designed to address study quality. The following areas were examined: representativeness of the targeted hospital staff, bias and confounding, description of the intervention, assessment of outcomes, and statistical quality and interpretation. The items in these categories were derived from study quality forms used in previous JHU EPC projects.^{8, 11} Items were modified to fit a focus on teaching strategies based on published criteria for evaluating an educational program.¹² The study team assigned each response level a score of zero (criteria not met), one (criteria partially met), or two (criteria fully met). The score for each category of study quality was the percentage of the total points available in each category and therefore could range from zero to 100 percent. The overall quality score was the average of the five categorical scores.

The content abstraction form was designed to collect such information as the description of the participants, the geographic location, the type of MCI, the training intervention, hospital staff targeted, and the hospital departments and other entities involved. The form also included items on the objectives of the training and the training evaluation methods. We classified objectives as knowledge, skills, behaviors, and clinical outcomes. On the form, we grouped outcomes and main conclusions of the drill by the target area involved in the exercise (e.g., incident command system, internal/external communications, patient flow and tracking, and security). We developed this grouping on the basis of discussions with experts and initial article review. This approach is consistent with the content of the job action sheets of the Hospital Emergency Incident Command System (HEICS), developed to assist the operation of a medical facility in time of crisis.¹⁰ Many hospitals have adopted the HEICS system as they develop disaster response systems.

Article Review Process

The EPC team conducted the article review in a serial fashion. The first reviewer completed the quality assessment form and the content abstraction form. The second reviewer then reviewed the article and checked each response on the forms. Any disagreements between the two reviewers were resolved by consensus. Reviewers were not masked to author or journal names because to do so is both costly and time-consuming, and previous work has shown that masking is unlikely to make a significant difference in the results of the review.¹³

Evidence Grading

For each question, the EPC team assigned evidence grades based on an established grading scheme with well-defined levels of evidence. The grading scheme, used in previous systematic reviews,^{14, 15} assigns grades as follows:

Grade A (strong): Appropriate data available for evaluating the outcomes of the training program, including at least one well-done randomized controlled trial; the population of learners is sufficiently large and well described, and adequate controls have been used; data are consistent; and the educational intervention is well described and one intervention is clearly superior, equivalent, or inferior to another for well-defined outcomes.

Grade B (Moderate): Appropriate data available for evaluating the outcomes of the training program; the population of learners is sufficiently large and well described, and adequate controls have been used; data are reasonably but not entirely consistent; and the educational intervention is well described and one intervention is superior or equivalent for well-defined outcomes, but there is insufficient evidence to make a definite conclusion of superiority of one approach over another.

Grade C (Weak): Some data for evaluating the educational intervention is available; the population is adequately large but poorly defined; there may be a trend for preference of one intervention over another for well-defined outcomes, but there is insufficient evidence to draw firm conclusions of superiority.

Grade I (Insufficient): Appropriate data not available, or there is an insufficient number of trainees to assess the intervention either alone or in comparison with alternatives.

Evidence Tables

Evidence tables were constructed to present the information addressing each key question. The evidence tables summarize the basic characteristics of each study, study quality, and results of the studies. Within each evidence table, studies are listed by type of training (i.e., hospital disaster drill, computer simulation, or tabletop and other exercises). The evidence tables are included in Appendix E.

Peer Review Process

The draft evidence report was sent to the 12 experts for peer review. Experts were asked to comment on the content of specific sections of the report according to their areas of expertise and interest. The EPC team addressed the reviewers' comments in the final report and submitted a detailed summary of the comments and responses to the AHRQ.