Suggested Improvements

Some workshop participants felt that, if case reports are to serve a useful function in assessing the safety of vaccines, there is a great need to improve the analysis of the case reports received in passive surveillance systems by improving the quality of the information contained within an individual case report, by harmonizing the systems that collect and analyze the data, and by increasing the proportion of serious events that are reported. The enthusiasm for better utilizing case reports was not, however, universal. Those with reservations worried that increasing reporting will merely increase the noise inherent in the systems and that the resources needed to improve passive surveillance systems sufficiently would be better spent on other research tools.

Improvements at the level of the individual case report have both quantitative (more reports and more information in each report) and qualitative (complete, relevant, and necessary information) aspects. A significant factor in the success of passive surveillance systems is whether the information about the reaction is complete and accurate. This is probably best achieved if a health care professional reports the reaction or verifies information received from a nonhealth care professional. The problem of underreporting of adverse events associated with vaccines is discussed in a subsequent section. Complete information on the vaccinee and the reaction to a vaccination are important. In addition to information on the patient's history and complete details of the medical problems with which the patient is presenting, it is important to know about other risk factors (for example, the possibility that an infectious disease is circulating within the community) that might account for the reaction. As mentioned previously, the absence of an alternative etiologic candidate is one factor in support of causality. Results of a comprehensive examination are useful for diagnostic and treatment purposes (What is wrong with the patient and what treatment should the patient be given?) and for research purposes (Did the vaccine cause the adverse event? If so, what is the mechanism?). A complete workup could be expensive, and it is not clear who should pay for a comprehensive medical examination of someone experiencing an adverse event after vaccination.

The analysis of adverse events could be greatly improved by harmonizing of the adverse reaction reporting forms, the coding terminologies, and the search strategies used to summarize the information contained in the multiple databases managed by different manufacturers and different countries. In September 1993 FDA sponsored a workshop on harmonization efforts, and the participants were hopeful that the workshop marked the beginning of more long-term efforts in that direction. Harmonization is an issue for manufacturers (How much effort is made to obtain complete data? How is the documentation organized? What coding system and coding practices are used?); for regulatory agencies (agencies may not use the same coding systems as the manufacturers); and for data analysis and report generation (e.g., How many cases of adverse event x have been reported with vaccine y?). The number of reports of a given adverse event identified within a system depends on the completeness of the search strategies and the categories used to classify the adverse event. Many errors can be made if search strategies are not standardized and validated. If the main purpose of passive surveillance systems is a signal detection system (particularly for the detection of previously unrecognized adverse events but also for the detection of increased numbers of reports of a recognized adverse event), participants suggested that search strategies should err in the direction of false positives rather than false negatives.

Data Analysis

The use of summary statistics (e.g., how many reports of reaction x after vaccine y) from passive surveillance systems is a first step in assessing the safety of vaccines, but limiting the analysis to those statistics can lead to erroneous conclusions. The reported events occur in temporal relation to vaccination, but not all events that occur within an appropriate interval from the time of vaccination are necessarily caused by the vaccination. Although this might seem obvious, summary statistics of reported events can be misinterpreted as evidence that the vaccine truly causes the adverse event.

As a passive surveillance system, VAERS suffers problems of both underreporting and overreporting. Although health care professionals are required to report some adverse events, specifically, those that are covered by the no-fault component of the Vaccine Injury Compensation Program, there are no provisions for enforcement. It is likely that many events that occur after the receipt of vaccines, like those that occur after the receipt of medications, go unreported. People may fail to report an adverse event because they are unaware of the reporting system, because they do not believe that the event is related to the vaccine, or because they know that the adverse event is an accepted reaction (e.g., fever or sore arm) to that vaccine and consider it unnecessary to report what is already established. Because anyone can report an adverse event to VAERS, it is possible that the patient, the manufacturer, and the health care provider could report the same event. If this duplication is not recognized and accounted for by the system, the number of adverse events in the system would be spuriously high. Duplication is not the major source of overreporting. Serious adverse events that occur after a vaccination may be reported even if the reporting physician knows or strongly suspects another cause.

In addition to over- and underreporting, it is difficult to assess the meaning of absolute numbers of reports for several reasons. Consider the following examples:

• Many people who experience an adverse event have more than one symptom and each symptom is given a separate code. A child who experiences a fever, a sore arm, and vomiting is listed as having three distinct adverse events. Thus, the number of adverse events recorded by the system is far greater than the number of people who have experienced an adverse event.
• One would expect more reports of adverse events following receipt of diphtheria and tetanus toxoids and pertussis vaccine (DPT) vaccine than following receipt of measles, mumps, and rubella vaccine simply because each child receives more doses of DPT than measles, mumps, and rubella vaccine.
• One would expect to find more reports of sudden infant death syndrome (SIDS) associated with a lot of vaccine in use in the winter months than with lots in use during summer because the background rate (and therefore the expected number of coincidental reports) of SIDS is higher in winter than in summer.
• Childhood vaccines are frequently given in combination, and each adverse event occurring following receipt of each vaccine given on the same day is given a separate code. Summary statistics from a passive surveillance system such as VAERS will show a great many reports of a sore arm following receipt of the oral polio vaccine (OPV). What that summary statistic does not indicate is that the child concurrently received other vaccines that were responsible for the sore arm. In fact, there is no causality assessment made for the reports in VAERS, because the system was not designed to support such an assessment.

These examples describe the expected complications of passive surveillance systems. Reporting and analysis of summary statistics require careful consideration of and explanations for these factors.

Signal Detection and Hypothesis Generation

Passive surveillance systems can be used to establish patterns of adverse event reporting and to monitor changes in those patterns as vaccines change (different lots, different formulations, different antigens), or as the populations that receive particular vaccines change. Although case reports can provide important information regarding causality under specific circumstances, the participants generally agreed that passive surveillance systems best serve to detect signals or warnings that there might be a problem rather than to answer questions about causation. These signals can lead to hypotheses about causality, which can then be tested by other methods, such as epidemiologic or laboratory studies.

A participant noted that because VAERS is intended to detect new signals, it would be inadvisable to impede that process by providing case definitions to those who are doing the reporting. He suggested, instead, the creation of standardized protocols, so that when a report falling into a particular category is received, a follow-up protocol is sent to that reporter for completion. The question that then arises is how to finance the extra workup that may be required, especially if it includes costly medical tests.

Participants expressed a desire for a more clearly defined mechanism by which passive surveillance systems are evaluated and hypotheses are generated. FDA and CDC routinely review VAERS reports in a search for signals, and they are still trying to determine the best method for doing this. Internal review of current reports looks for patterns of events as well as the overall numbers of reports. Summary data from the VAERS system have not routinely been published, but action is under way to determine the best format for doing so. It was suggested that the publication of well-documented case reports should be encouraged and facilitated.

More rigorous, well-established epidemiologic and statistical methods could be applied to passive reporting systems, and PHS representatives indicated that some such methods are under development. For example, in the fall of 1993, a joint meeting of FDA and CDC investigators looked at clusters of adverse events, particularly deaths associated with specific vaccine lots. Although analysis of such clusters can be useful, it usually does not permit inferences about causality. Again, it serves to sound an alert that there might be a problem, that then needs to be analyzed in some other way. It was suggested that VAERS data are amenable to time series analysis; outcomes can be examined across time (by month, by 6-month period, etc.) and can be used as systematic signal-generating mechanisms.

A participant expressed caution in interpreting the information from passive surveillance systems, noting that “one person's signal is another person's noise.” If all the signals are preserved, there is so much attendant noise that undue fears about the safety of vaccines may be raised. That participant felt that various signals need to be preserved and made readily accessible to the people who are generating and testing hypotheses, but urged that care be taken to avoid misinterpretation of what common scientific wisdom would suggest is uninterpretable noise as a signal of a causal relation. Such spurious associations could unnecessarily alarm the public and could lead to decreases in immunization coverage.

There was discussion of how much money and effort should be devoted to passive surveillance systems given their inherent limitations. One participant expressed the opinion that financial resources should be allocated on the basis of the potential benefit of a particular research method. That participant preferred an emphasis on allocating tight monetary resources to rigorous scientific studies once the alert has been sounded. However, scientific understanding of the mechanisms of adverse reactions to vaccines has not yet advanced to the point that it is always evident what studies should be undertaken.

Use of Case Reports in Causality Assessment

One participant stressed that individual case reports from passive systems may in some specific cases, be able to answer causation questions that population-based studies never could. For example, the question of whether the Urabe-strain mumps vaccine can cause meningitis or whether OPV can cause polio can be answered by a single persuasive case in which virus is isolated and is typed by molecular biologic techniques as the vaccine strain, rather than the wild-type strain, and there is no other etiologic factor that could cause the disease. The absence of an alternative etiologic factor makes the isolation of the vaccine strain of the virus more compelling, because the presence of vaccine strain virus does not always prove that the virus caused disease; it is not known how often vaccine-strain virus might be present in asymptomatic individuals.

Several participants agreed that evaluation of case reports of fully evaluated events should not be minimized as an important method of examining very rarely occurring adverse events. Even LLDBs do not have access to data on populations in the millions, such as might be necessary to have enough power to make conclusions about rarely occurring adverse events in a population. When a suspected rarely occurring adverse event is reported following vaccination, intense and targeted clinical evaluations and research might help in assessing causality. As discussed above, this could be costly, and currently there is no mechanism to fund these investigations.

Some shortcomings of passive surveillance systems are common to other strategies for evaluating adverse events; for example, no data-gathering system, including LLDBs, can easily assess causation when multiple vaccines are administered simultaneously. As another example, because of the emphasis on events temporally associated with vaccination, events that occur long after vaccination may not be detected.

Because causation may not be the most fruitful focus for a discussion of VAERS and other passive surveillance systems, one participant suggested that two additional questions be asked.

1.

Do preexisting conditions play a role in the occurrence or reporting of adverse events?

2.

Are particular lots associated with higher reporting rates of particular adverse events?

Passive surveillance systems such as VAERS could be helpful in addressing both of these questions. However, as with questions of causality, without complete information in each report, these questions are unanswerable by a passive surveillance system.

Education of Health Care Providers

A participant noted a problem in persuading doctors and other health care providers to report adverse events. Health care providers need to be educated that VAERS is an adverse “event” reporting system, not an adverse “effect” reporting system. That is, physicians or other health care providers sometimes fail to report an event after vaccination because they do not believe that the event is related to vaccination. This tacit decision to report only well-established adverse consequences of vaccines means that the previously unrecognized effects of vaccines would continue to go unnoticed. If a passive surveillance system is to function effectively as a signal detection system, then all serious adverse events that occur following vaccination should be reported.

A participant noted that, with the advent of the new presidential administration, immunization policy in general is being reevaluated. The education of practitioners, including information about adverse events and the provision of instructions on how to report them, is expected to be an integral part of a new immunization program.

Vaccine Injury Compensation Program

A participant noted that the Vaccine Injury Compensation Program (VICP) also serves as a passive reporting system that might supplement the information in VAERS. There is potentially more useful information in the records from VICP than in those from VAERS, because vaccinees or their parents are required to submit complete medical information to be considered for compensation. In addition, there could be cases in the VICP that are not in VAERS. For example, the participant noted a case submitted to VICP (but not to VAERS) of two episodes of Guillain-Barré syndrome (GBS) in the same child after administration of a tetanus toxoid-containing preparation. The possibility of linking the VAERS and VICP databases is under active discussion within PHS. Another participant noted that databases from Japan might be informative, for example, for providing information on that country's experiences when it changed from recommending DPT vaccination in infancy to age 2 years and older.

Evaluation of Passive Surveillance Systems

An evaluation of VAERS itself and how it has functioned thus far was suggested. It should be determined, for example, whether VAERS actually has served well as the signal detector that a passive surveillance system is intended to be. Research to compare surveillance systems was suggested—to find out, for example, what proportion of cases of various sorts are reported to VAERS. This would require comparison with LLDBs and with active surveillance systems to see what proportion of cases are reported to each and how that proportion changes with the type and severity of adverse event reported. Some existing research on statistical modeling of adverse event reporting systems shows that relative risks must be very high (given the state of underreporting) or the adverse event extraordinarily unusual for passive reporting systems to detect adverse events.

Individual case reports, if well-reported, have a role in assessing causality under specific circumstances. There was considerable disagreement about the value of larger investment in passive surveillance systems for studying the safety of vaccines. There was call for further consideration of when case reports are useful, how they are useful, how to improve them, and how to make sure that cases are reported and that reports are disseminated.