Prioritization Models for New Vaccine Development

The IOM has contributed scholarly work to the subject of vaccine prioritization since the 1980s. In 1985 and 1986 the IOM published two reports under the same title, New Vaccine Development: Establishing Priorities, one that focused on vaccine priorities for the United States (IOM, 1985) and another that focused on international priorities (IOM, 1986). These two reports used equivalents of infant lives saved as the sole measure of benefit in prioritizing vaccines. IOM's next prioritization report, Vaccines for the 21st Century (IOM, 2000), used an efficiency measure (in the form of a cost-effectiveness ratio) rather than a direct benefit measure such as life-years saved and focused only on U.S. vaccine priorities. The approach used in Ranking Vaccines: A Prioritization Framework (IOM, 2012) and in this report has been informed by these previous studies but has significantly expanded the attributes that are relevant for the prioritization and development of new vaccines.

The committee, which gathered feedback from groups and individuals with a broad range of perspectives about the 1985–1986 and the 2000 reports, regularly heard that while those reports were valuable, their focus on life-years saved and cost-effectiveness ratios as outcomes limited their usefulness. The Phase I committee thus chose an approach that provided users and stakeholders with a list of vaccine attributes and allowed them to choose which particular ones they would use in prioritizing vaccines. The committee also developed an intuitive approach to prioritizing vaccines, based on the multi-attribute utility model, which allows users to develop prioritizations based on their assessments of the relative importance of various attributes. One important advantage to the approach developed by the committee is that while life-years saved, cost-effectiveness ratios, and other traditional measures are still available for selection, SMART Vaccines allows users to evaluate vaccine candidates based on many other important attributes, including, for example, the benefits to vulnerable populations or the potential to improve production platforms or delivery methods.

After Phase I of the project had been completed, the NVPO commissioned the IOM to collect feedback and to continue the development of SMART Vaccines, which the second committee has done, taking into account information obtained from a public workshop and from the presentations of various committee member carried out at the beginning of Phase II. The committee's task, as laid out by the NVPO, also included expanding the vaccine datasets to include at least three more vaccine candidates. In addition to the influenza, tuberculosis, and group B streptococcus vaccines tested in Phase I, the committee included human papillomavirus, pneumococcal infection, and rotavirus as test vaccine candidates for both the United States and South Africa (the same two countries considered in Phase I) in this study. Box S-1 provides the complete charge to the committee for its Phase II work.

BOX S-1

Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase II. Engage stakeholders and obtain feedback about the usefulness of the prototype model developed in Phase I for prioritizing new preventive vaccines for development. (more...)

Design of SMART Vaccines

SMART Vaccines 1.0 is a decision-support tool that is intended to help users carry out more effective discussions and make better decisions about the research and development, manufacturing, implementation, and delivery of vaccines. It provides a scientific basis for decision making in an environment characterized by financial pressures, uncertainty, and a lack of standard information. Decision makers and other users can employ SMART Vaccines to assist them in reaching a consensus decision or simply to guide them in establishing the knowledge base needed in various decision-making scenarios. A particularly useful characteristic of SMART Vaccines is that it offers dynamic capabilities that allow users to examine several scenarios by changing the inputs and seeing the results change instantaneously.

SMART Vaccines has four basic inputs: (1) the demographics of the population to be immunized, (2) the disease burden for that population, (3) the value-relevant attributes of potential vaccines, and (4) the user's ranks and weights relating to the vaccine attributes.

Demographic Characteristics

The user specifies the population of interest (e.g., a nation, a state or province within it, or perhaps a consortium of nations) and then imports life-table data for that country or region from World Health Organization (WHO) databases. The user can also focus on specific populations with special characteristics, such as infants less than one year old or HIV-positive individuals.

Disease Burden

Next, the user specifies the disease and enters data for the associated disease burden. The required data include incidence rates (by age and sex), case-fatality rates (by age and sex), morbidity due to the disease, duration of the condition, how health-related quality of life is affected by the condition, and the estimated costs associated with treatment of the disease. Data from WHO and the Global Burden of Disease project can be used to provide a basic source of information for nations that do not have a reliable disease surveillance system.

Vaccine Characteristics

The user next estimates the economic and functional characteristics—for example, potential efficacy, uptake, and product development costs—of the candidate vaccines. Many of these characteristics will be unknown, especially for new or undeveloped vaccines. Therefore, SMART Vaccines allows the user to explore how sensitive the ultimate rankings are to the various parameter estimates.

Attributes and Weights

This portion of SMART Vaccines is its most novel feature. Users can select from 28 attributes arranged in eight categories. The committee chose to retain the original attribute list following the discussions with various stakeholders during Phase I, with an addition of a ninth category of user-defined attributes, which allows up to seven qualitative attributes defined by the user (see Table S-1). Attributes concerning health and economic considerations are calculated by the computational submodel with the provided data while the remaining attributes are value preferences selected by the user. In particular, the user-defined attributes are qualitative binary assessments requiring a “yes” or “no” response. Users specify which of the 28–plus attributes will be considered in the multi-attribute utility function and place those attributes in rank order, the first being most important.

TABLE S-1

Choices of Attributes in SMART Vaccines 1.0.

SMART Vaccines approximates a set of weights for the rank-ordered selections by using a mathematical process known as the rank-order centroid method. This method calculates averages of all weights and assigns weight to each attribute corresponding with the user's rank order, with the final weights adding up to 100 percent. Most of the weight is placed on the first five to six attributes. In the committee's experience, most applications of multi-attribute utility theory (whether using the rank-order centroid method or not) place only small weights (5 percent or less) on attributes that are ranked below the fifth attribute. The weight of each attribute beyond the seventh one becomes very small (less than 2 percent). Although SMART Vaccines 1.0 allows users to select up to 10 attributes, selection of no more than 7 attributes is suggested in order to allow each weight to sufficiently influence the SMART Score However, should the user wish, the weights applied to each attribute can be adjusted with slider bars, altering the weight positions until the user is satisfied with the final weights applied to each attribute for every vaccine candidate under consideration.

SMART Score

The multi-attribute utility model underlying SMART Vaccines produces a value score—called a SMART Score—which helps users interpret the relative performance and rank of their vaccine candidates. Because each user will have specified his or her own value function, a sample SMART Score of 60 has meaning only when comparing other vaccines within the same user's framework. Comparisons across individual users' scores are meaningful only if the users select the same attributes and use identical endpoints (worst and best level) for each attribute. Otherwise, a score of 60 for one user may mean something very different than a score of 60 for another user.

Multi-attribute utility scales preserve the order of preferences or priorities among vaccines. A vaccine with a higher score is preferred or has higher priority. SMART Scores are always relative to the user's choice of two reference points: a zero (assigned to a vaccine that is the worst possible on all attributes) and a score of 100 (a vaccine that is the best possible on all attributes). Thus, a score of 50 would mean that the vaccine is halfway between the worst and best vaccines. It is also meaningful to interpret differences between SMART Scores. For example, a difference of, say, 10 points has the same meaning anywhere on a single user's scale, so a difference of 10 points on a SMART Score between 60 and 70 has the same meaning as a difference between 30 and 40. However, it is not correct to think of a vaccine with a score of 40 as being twice as good as a vaccine with a score of 20.

One way to understand this is in analogy with measures of temperature. Some thermometers measure temperature in Celsius, some in Fahrenheit, and some in Kelvin. Only the Kelvin scale begins at absolute zero, so it alone allows statements about the relative magnitude of its values—300 K is twice as hot as 150 K, while 20°C is not twice as hot at 10°C. Comparisons of temperatures across these scales lack intrinsic meaning unless each user has a “standard event” he or she can measure. With thermometers, the values for the freezing (32°F and 0°C) and boiling (212°F and 100°C) points of water provide such measures. Knowing these two “standard scores” allows us to also understand that 20°C is the same temperature as 68°F. Final scores from different users cannot be aggregated to obtain a common SMART Score because the users may have chosen a different set of attributes for their valuation; hence each score reflects different priorities based on different preference structures. But, users can help calibrate each others' SMART Scores by ranking two or more vaccines where the population, disease burden, treatment cost, and vaccine attributes are all identical (e.g., comparing vaccine candidates in the United States for influenza and tuberculosis). This comparison is most fruitful within a single population, for instance, comparing influenza and tuberculosis for South Africa is useful whereas comparing a new influenza vaccine for the United States against a new tuberculosis vaccine for South Africa is infeasible. This is because the disease burden and other characteristics in South Africa differ greatly from those in the United States for both influenza and tuberculosis, thus, the comparison across two populations is not a valid one.

Test Vaccine Candidates

Building upon the Phase I work, the Phase II committee chose to add three additional vaccine candidates for use in evaluating the software and to expand the data library for SMART Vaccines. Again, the United States and South Africa were chosen as the populations representing the different perspectives of high- and low-income nations and also to provide an interesting contrast in disease burdens. In addition, these two countries also have different income, health, and demographic profiles, and different social and economic priorities for developing and delivering vaccines.

The previously existing portfolio of diseases consisted of influenza, tuberculosis, and group B streptococcus. The committee chose to add human papillomavirus, pneumococcal, and rotavirus vaccines as the three additional test candidates. Human papillomavirus infects individuals when they become sexually active and may progress to cervical or anal cancer with time. Both rotavirus and pneumococcal infections occur commonly in children and have a greater impact in low-income settings.

Data Needs

SMART Vaccines is only as robust as the data that are available for use in its calculations. However, the data gathered by this committee for the test case vaccines are only estimates, intended to demonstrate the functionality of SMART Vaccines. The data were gathered to provide a starting point for users to edit or change the data or to introduce their own information. And if users wish to compare vaccines other than the ones provided with the software, they can modify the pre-loaded data as necessary.

Many different types of data, including data on demographic factors, disease burden, economic factors, and vaccine characteristics, are available to users from various sources and estimations. Because many vaccines are still hypothetical—which means that data about them do not yet exist—a user who wishes to analyze such a vaccine must provide some estimates of what could be possible (such as cost per dose, the developmental costs for the vaccine), and these estimates may be difficult to determine. Much of the remainder of the information needed to use SMART Vaccines can be obtained from public sources and the published literature, but the data vary in comprehensiveness and accuracy.

The next stage of this study—Phase III—will attempt to provide estimation strategies toward assisting users in thinking about data compilation for SMART Vaccines. In doing so, an Institute of Medicine committee is also expected to collaborate with potential users to determine software use case scenarios.

Accessing and Using SMART Vaccines

An executable file of SMART Vaccines 1.0, currently available for computers running the Windows operating system, can be downloaded from the Institute of Medicine (IOM) website (www.iom.edu/SMARTVaccines) or from the National Academies Press (www.nap.edu/SMARTVaccines). The current version is pre-populated with test data which allow the user to use the model to evaluate vaccine candidates chosen by the committee for the United States and South Africa. To compare vaccine candidates other than the ones provided in the software, users will need either to import data from other sources or to provide their own data. The software leads users through this process to some extent, generally relying on users to first enter data in spreadsheet format (e.g., using Microsoft Excel). These data can then be imported into SMART Vaccines.

To assist individuals in using SMART Vaccines, spreadsheets containing data for the six vaccine candidates have been made available along with an empty spreadsheet template that is included for data entry purposes, if needed by the user, and that is available on the same websites where the software and this report are available for download.

Next Steps

As has been noted in Ranking Vaccines: A Prioritization Framework (IOM, 2012) and re-emphasized in this report, SMART Vaccines should not be thought of as a decision maker. It is a decision-support tool intended to provide insight to users and to facilitate discussions before ultimate decisions are made.

To inform future versions of SMART Vaccines, the committee adopted a guiding principle: SMART Vaccines will have the greatest potential and value if it is programmed as a dynamic, continuously evolving software application, and made freely available in an open-source environment to all decision makers and developers around the world.

The committee also believes, as a related strategy, that the benefit will be achieved with the greatest likelihood if the National Vaccine Program Office of the Department of Health and Human Services identifies a host for SMART Vaccines and its future versions. Furthermore, no decision-support system, including SMART Vaccines, has any intrinsic value without accurate and relevant data. Consequently, the committee places highest importance on the creation of a data architecture and expanding data collection for use in SMART Vaccines.

This last point in turn leads to six related future conditions that the committee believes will enhance the long-term success of SMART Vaccines. These conditions are:

1.

SMART Vaccines—and its future versions—is hosted in an open-source setting on a widely trusted website with a distinct identity and is appropriately protected from unwarranted modifications or intrusions.

2.

The host organization creates, maintains, funds, and facilitates a community of users to curate and manage further development of SMART Vaccines and supporting data.

3.

The community of users includes decision makers involved in research, development, regulation, and implementation of new vaccines as well as developers with expertise in such areas as modeling, epidemiology, demography, software engineering, database management, and visual design.

4.

The community of users—independently or in collaboration with the host organization—establishes an advisory group to help plan future versions and the adoption of SMART Vaccines.

5.

The community of users, together with the host and sponsors, develops mechanisms to encourage the development and updating of various types of data: on populations at regional, national, and sub-national levels; on the disease burdens they confront; on the costs of preventing and treating those diseases in each distinct environment; and on the productivity losses associated with these disease burdens. Ideally, these data are accessible in a standardized format, shared with other users through the common website that hosts SMART Vaccines, and improved through an editing process agreed upon and overseen by the user community. These processes could ultimately help guide improvements in global communication and coordination of data and initiatives of common interest and shared importance.

6.

The community of users studies the application of SMART Vaccines for retrospective analysis, validation, or confirmation of previous decisions relating to new vaccine development. Results would have both an educational and a continuous learning benefit.

Immediate next steps for further development of SMART Vaccines could focus on creating a data warehouse that enables users to create, share, access, and validate data for a broad range of populations, diseases, and vaccine candidates in standardized formats. Without large increases in the availability of structured data, it will not be possible to create prioritization catalogues. A data warehouse of this sort could be seeded with publicly available population data, and it would likely be focused on nation-level statistics, but it could also include global, regional, or state-level data as required by the user base. Others—for example, vaccine manufacturers—may wish to take a more global perspective but with a narrower set of candidate vaccines. Another important next step would be user review of the software design, coupled with formal usability studies targeted at potential user organizations, in order to develop a flexible software design that will ensure that SMART Vaccines is maximally intuitive for a broad range of end users and easily extensible by the open source community.

Observations

The study described in this report is a wholly novel exercise for the IOM and the National Academies in that a primary output of the committee's work is a software product. Developers and users of any commercial software understand that keeping software current requires continual improvements and upgrades. No software application is flawless in its first version. SMART Vaccines is no different, and, moreover, it has been developed in an academic and policy setting rather than in an industrial software engineering environment. For these reasons the committee has set forth a vision to carry the work of SMART Vaccines forward, both in database development and in software enhancements through usability studies and other strategies.

Unlike previous IOM reports on vaccine prioritization, this study does not provide a “list” of vaccine priorities, nor was the committee tasked with achieving such an outcome. Users of SMART Vaccines will create their own priority lists with their own values and available data. In short, rather than imposing the value system of the committee, SMART Vaccines allows users to specify what is most important to them—thus creating their very own value structures, each of which will result in its own unique list of vaccines.

If appropriately used, SMART Vaccines should help enhance discussions among users about differences in their priority lists and about the explanations for those differences. The committee hopes that SMART Vaccines will allow every voice to be heard in such discussions.