The Decision-Making Environment and the Importance of Process

Many decision-making situations involving matters of public heath and environmental risk have five common elements: the desire to use the best scientific methods and evidence in informing decisions, uncertainty that limits the ability to characterize both the magnitude of the problem and the corresponding benefits of proposed interventions, a need for timeliness in decision-making that precludes resolving important uncertainties before decisions are required, the presence of some sort of tradeoff among disparate adverse outcomes (which may be health, ecologic, or economic outcomes, each affecting a different set of stakeholders), and the reality that, because of the inherent complexity of the systems being managed and the long-term implications of many decisions (such as cancer latency, changes in the structure of ecosystems, or multiple simultaneous sources of exposure), there will be little or no short-term feedback as to whether the desired outcome has been achieved by the decisions.

The combination of uncertainty in the scientific data and assumptions (the “inputs”) and inability to validate assessment results directly or to isolate and evaluate the impact of a resulting decision (the “outputs”) creates a situation in which decision-makers, the scientific community, the public, industry and other stakeholders have little choice but to rely on the overall quality of the many processes used in the conduct of risk assessment to provide some assurance that the assessment is aligned with societal goals.

Those challenging properties of the decision-making environment may be considered particularly acute for many health and environmental decisions, but they are by no means new to decision-makers generally. The academic discipline of decision analysis under uncertainty, among others, has a rich literature on which to draw for methods and findings (Morgan et al. 1990; Clemen 1996; Raiffa 1997). The importance of attention to process is entirely compatible with the theory of the management sciences that defines a good decision under uncertainty as one that uses the most appropriate processes and methods to assemble and interpret evidence, to apply the decision-maker’s values properly, and to make timely choices with available resources rather than defining a good decision only according to its (apparent) outcomes. This attention to process is also compatible with arguments for the inclusion of more deliberative approaches to assessment and decision-making. As such, the most appropriate processes and methods in a given situation may be an appropriate balance of deliberative and analytic methods, as advocated in NRC (1996).

Risk Assessment as a Decision-Support Product

The process of risk assessment involves generation of a number of individual products that are combined to form a final product (which is often referred to as “the risk assessment”). The final product of a risk assessment process is most often understood to be a report. The present committee suggests that the product of a risk assessment should be considered to include not only the report but various subproducts, such as computational models and other information that is assembled during the process. The subproducts have different uses and serve a variety of audiences. For example, a computational model with a user-friendly interface may be at least as valuable in informing decision-making as the technical report most often associated with the term risk assessment. In addition, such subproducts as dose-response assessments typically have value that transcends a particular decision-support application and may be used in thousands of future decision-support situations. It is also useful to consider that risk assessments and individual subproducts experience a life cycle (consisting, for example, of conception, design, development, testing, use, maintenance, obsolescence, and replacement) that should be explicitly recognized.

The products of risk assessment may be thought of as, among other things, communication products. Their value lies in their contribution to the objectives of the decision-making function, including their effects on the primary decision-maker and other interested parties who participate in the decision or otherwise use the information that the products convey. Although the effort expended in the process is largely scientific, the critical final process in risk assessment is ultimately communication.

The Quality of Risk Assessment Includes Both Process and Product Attributes

The decision-making environment associated with health and environmental risk management compels the various users of risk assessment to value and scrutinize the assessment process. In addition, risk assessment is understood to result in a set of final products whose specific attributes are critical for meeting their objectives. In a sense, it may be neither possible nor appropriate to separate the process from the product. The situation is somewhat analogous to that of other products whose quality is more readily scrutinized with respect to the process that is used rather than through scrutiny of detectable qualities of the final product. For example, the safety aspects of the quality of complex engineered systems, medical devices, and foods are increasingly scrutinized with respect to the quality of the process that generates and maintains them rather than judged solely on the basis of measurable qualities of the final product. Similarly, the final products of a risk assessment have a mixture of detectable and undetectable qualities, and both the final product and the underlying process must be considered in judging the overall quality.

Given the demands of health and environmental decision-making, perhaps the most appropriate element of quality in risk-assessment products is captured in their ability to improve the capacity of decision-makers to make informed decisions in the presence of substantial, inevitable and irreducible uncertainty. A secondary but surely important quality is the ability of the assessment products to improve other stakeholders’ understanding and to foster and support the broader public interests in the quality of the decision-making process (for example, fairness, transparency, and efficiency). Those attributes are difficult to measure, and some elements of quality often cannot be judged until some time after the completion of the risk assessment.

Formative and Iterative Design of Risk Assessments

For the committee’s purposes, the term design implies adopting a user-centered perspective to craft both an assessment process and a decision-support product that achieves the objectives of supporting high-quality decision-making while working within inevitable constraints. Accordingly, an important part of the early design process is the understanding and weighing of all the objectives, recognition of constraints, and explicit acknowledgment of the need for tradeoffs.

Design will inevitably occur throughout the risk-assessment process, and flexibility and iteration will be important aspects of the overall process design. Like any complex product designed in a complex environment, the process and product may need to be redesigned as objectives and constraints inevitably change and in response to new knowledge. While recognizing the iterative nature of risk-assessment planning, the committee strongly encourages increasing attention to design in the formative stages of a risk assessment. Such a shift in attention is recognized by the Environmental Protection Agency (EPA 2004a). It is also captured in guidance documents for ecologic risk assessment and cumulative risk assessment (EPA 1992, 1998, 2003). In those applications, EPA has adopted two tasks labeled planning and scoping and problem formulation. The two tasks are examples of early design activities, and the committee believes that they should be formalized, applied more consistently in risk-assessment activities, and, perhaps most important, result in concrete outputs detailing the rationale and findings of the early design process. The tasks are described in more detail later in this chapter.

Objectives Related to Inputs

Inclusiveness of Scope

For various reasons, human health risk assessment has traditionally focused on single cause-effect pathways that involve a single chemical and single identified adverse effect. The narrowness of scope is frequently questioned with respect to both its scientific merits and its relevance to decision contexts of considerably greater scope. The scope of consideration in health and environmental risk management would ideally be as large as possible. It can be argued that any limitation in scope constitutes a simplification of reality that must be recognized and justified because important parts of the total cause-effect network may have been missed. A narrow scope has the potential to distort the external validity of the conclusions and the associated decisions they support and thus to limit their applicability to the “real world.”

From a decision-support perspective, limitations in scope might create what is seen as highly imbalanced information support, supporting a particular concern with voluminous technical analysis while other concerns of great relevance to stakeholders (which cannot be readily dismissed on purely scientific grounds) remain largely or completely unaddressed and explained only by the chosen scope of the risk assessment. For example, in situations where stakeholders are concerned about exposure from both food and water pathways, the provision of an elaborate risk assessment for waterborne exposure while providing only a cursory review for foodborne exposure may appear to be imbalanced with respect to the information needs. A somewhat more simplified risk assessment that includes both pathways may be preferable, if the foodborne pathway cannot be dismissed on strong grounds. Here, the objective of broadening of scope may compete with the desire to perform the “best” risk assessment on a single pathway.

The desire to broaden the scope of human health risk assessment appears to be shared by EPA. Table 3-1 illustrates the expansion of the scope (in both risk assessment and decision-making) to which EPA aspires, at least as far as can be inferred from its guidance for cumulative risk assessment. Some of the “new” characteristics are current practice in ecologic risk assessment.

TABLE 3-1

Transition in EPA Human Health Risk-Assessment Characteristics According to EPA (1997).

A critical dimension of scope (and a theme of Chapter 8 of this report) is the explicit inclusion of the various possible mitigation options that might be considered to reduce the risk that is being assessed. The scope would be expanded so that the assessment would provide not only estimates of existing risk but estimates of risk reduction associated with a variety of changes in the risk-generating system. To provide more complete information to the decision-maker, the decision-support products would ideally include (or be reasonably integrated with) estimates of the associated costs and any countervailing risks associated with the proposed mitigation options, as might be presented, for example, in a remedial action report under Superfund or in assessments that inform pesticide registration decisions.

Additional elements of scope derive from the desire to support decision-makers other than EPA’s internal risk managers. The often-advocated goal of supporting local decision-makers, communities, and industrial stakeholders in a participatory decision-making model suggests the need for more customized decision-support tools on the basis of the nuanced information needs and value foci of other decision-makers. This implies either that the scope of the risk assessment increases to include those diverse needs and values or that separate assessments are conducted with different scopes and end points considered (with the associated problems of compatibility).

The concept of extended decision support can be taken further to support the broad array of decisions that EPA may not be directly involved in but ultimately is interested in their being risk-based, particularly for preventive risk management. Product and process- development decisions that are made every day around the world and have short-term and long-term effects on human health and the environment may be the most important class of external decisions that would ideally be increasingly risk-informed. This class of decisions includes decisions based on life-cycle analysis and various related approaches with similar goals, in which risks are ideally reduced by design of energy and material flows in advance rather than by end-of-pipe mitigation strategies. Some of these preventative strategies may benefit from risk-assessment components (like dose-response information, or quantification of common exposure scenarios) without the need for an entire risk assessment to be completed. This might suggest that risk-assessment products be designed, prepared, and disseminated in a modular fashion to allow for the individual components to be used and reused by third parties making different types of decisions.

Objectives Related to Process

Objectives Related to Impact on Decision-Making

Planning and Scoping

In 1989, EPA’s guidance for Superfund provided several pages of guidance specific to the planning and scoping of a human health risk assessment (EPA 1989). Because assessment of complex ecologic systems challenged both decision-makers and assessors, it was the ecologic risk-assessment community that ultimately championed the need to define the scope of a risk assessment and the need for discussion between decision-makers, assessors, and interested parties from the outset of an assessment. The need to scope an assessment and the need for assessors and managers to interact were discussed briefly in EPA’s 1992 framework for ecologic risk assessment (EPA 1992). NRC (1993) advocated for the integration of ecologic risks into the 1983 Red Book paradigm, and expressed a need to extend this paradigm to include the need for interaction between risk assessment and management at the early stages of a risk assessment, based on experience in ecologic assessment. In 1996, a National Research Council committee commented on the importance of planning from the beginning of a risk assessment (NRC 1996). In 1998, EPA released its guidance for ecologic risk assessment, which superseded the 1992 framework document and provided a greatly expanded discussion of scoping and of the roles of assessors and decision-makers; it also drew a clear distinction between the goals and content of the planning and scoping stage and the problem-formulation stage. More recently, EPA has further articulated how critical planning and scoping are for the conduct of a successful risk assessment and has provided detailed guidance for their conduct (EPA 2003, 2004a). During planning and scoping, a team of decision-makers, stakeholders, and risk assessors identifies the issue (or concern, problem, or objective) to be assessed and establishes the goals, breadth, depth, and focus of the assessment. Once the decision to use a risk assessment has been made, this stage becomes critical for developing a common understanding of why the risk assessment is being conducted, the boundaries of the assessment (for example, time, space, regulatory options, and impacts), the quantity and quality of data needed to answer the assessment questions, and how decision-makers will use and communicate the results. During this stage, decision-makers charged with protecting health and the environment, in the context of other competing interests, can identify the kinds of information they need to reach their decisions, risk assessors can ensure that science is used effectively to inform decision-makers’ concerns, and stakeholders can bring a sense of realism and purpose to the assessment. This stage is a focal point for stakeholder involvement in the risk-assessment process and the point at which risk communication should begin (EPA 2003). The relevance of risk-assessment results to decision-making can be enhanced by the up-front involvement of decision-makers and stakeholders in setting goals, defining options, and defining the scope and complexity of an assessment (Suter et al. 2003). Together, all can evaluate whether the assessment will help to address the identified problems (EPA 2004b).

While a common plan for the risk assessment is one of the goals of these stages, reaching consensus on all aspects of the scope and conduct of a risk assessment among decision-makers and stakeholders representing diverse interests, will not always be feasible. In addition, it is not necessarily in the public interest to delay the risk assessment where consensus is difficult to achieve. The process requires a balance among the competing values of deliberative input into a risk assessment, timeliness in the risk assessment process, and the resource burden associated with these early stages.

Early Identification of Decision-Making Options

As discussed later in this chapter and further in Chapter 8, the utility of a risk assessment is greatly enhanced when it is constructed and carried out in the context of a clear set of options under consideration by the decision-maker. Figure 3-1 explicitly includes identification of options as a critical element of planning and scoping. Although present EPA guidance (for example, on ecologic risk assessment, cumulative risk assessment, and air toxics) does not contain exact language calling for the explicit identification of decision-making options during the planning and scoping stage, it does allow preliminary consideration of regulatory or other management options. Existing EPA risk-assessment frameworks unquestionably contemplate consideration of options as they are related to decision-making, with plenty of interpretive room for arraying options if that is desired by or available to decision-makers and risk managers. For example (EPA 1998, p. 10), “risk assessors and risk managers both consider the potential value of conducting a risk assessment to address identified problems. Their discussion explores what is known about the degree of risk, what management options are available to mitigate or prevent it, and the value of conducting a risk assessment compared with other ways of learning about and addressing environmental concerns” (emphasis added). Not every issue faced by a risk manager will necessarily lend itself to “arraying options.” Some complex problems may also best be addressed by completing a thorough assessment of health risks and vulnerable populations prior to considering necessary control options. The Clean Air Act has used this approach to reduce air pollution concentrations over the past four decades. In the management of contaminated sediment, for example, it may be possible to examine the tradeoffs between various options, such as removal vs monitored natural recovery or capping versus hot-spot removal; in the case of soil contaminants for which no practical treatment options exist, options may be limited to various degrees of soil removal; and there may be instances in which the regulatory environment is so prescriptive as to preclude all but a few stipulated options.

Although the planning and scoping stage is primarily deliberative, in that it involves extensive discussion between decision-makers and stakeholders and to a smaller extent with risk assessors, it is expected to produce tangible products that are critical for the performance of a credible and useful risk assessment (EPA 2003, 2004a). The primary product is a statement, with explanation, of why the assessment is being performed and what it will include and exclude (that is, how comprehensive it will be). Other products may be descriptions of those involved and their roles (for example, technical, legal, or stakeholder advisers), key agreements made and understandings reached among those involved, the resources (such as budgets, staff, data, and models) required by or available to the assessment, and the schedule to be followed (including provision for timely and adequate internal and independent external peer review). A statement (Box 3-3) often summarizes the end result of the planning and scoping process, describing the specific concerns that the risk assessment will address and generally what will be included in its purview. The problem-formulation stage, whose specific products are a conceptual model and an analysis plan, develops the specific technical details for the assessment laid out during planning and scoping.

BOX 3-3

Planning and Scoping: An Example Summary Statement. “Air toxics emissions may be causing increased long-term inhalation health risk (both cancer and noncancer concerns) to people in the immediate vicinity of Acme Refining Company. A modeling risk (more...)

Problem Formulation

The extension of the concept of “problem formulation” to human health risk assessment first emerged during a 1991 National Research Council–sponsored risk-assessment workshop where the absence of such an activity in health risk assessment and the criticality of its use for ecologic risk assessment were discussed (NRC 1993). In 1992, EPA published Framework for Ecological Risk Assessment as the first statement of principles for ecologic risk assessments, including a further articulation of the concept of problem formulation (EPA 1992). The concept reached fruition in the agency’s 1998 Guidelines for Ecological Risk Assessment, which superseded the 1992 framework document (EPA 1998). Those documents describe methods for conducting conventional single-species, chemical-based risk assessments and techniques for assessing risk to ecosystems from multiple exposures (or stressors) and multiple effects (or end points) (EPA 1991). For several reasons, ecologic risk assessments in the United States have generally placed a greater emphasis on problem formulation than have human-health risk assessments (Moore and Biddinger 1996). But by emphasizing completion of problem formulation early, the ecologic-risk framework provides a clear procedural advantage over the existing human health risk framework in achieving an assessment that can be used to inform a management decision. The advantage is derived from having decision-makers and stakeholders as active participants from the beginning of an assessment rather than passively awaiting receipt of the results.

The problem-formulation stage sketches out the technical implications and decisions that are implied by the discussions that occur among decision-makers and stakeholders during planning and scoping so that risk assessors can proceed with the technical aspects of the assessment in a manner consistent with the decision context. This stage translates the results of the planning and scoping stage into two critical products: a conceptual model that explicitly identifies the stressors, sources, receptors, exposure pathways, and potential adverse human health effects that the risk assessment will evaluate and an analysis plan (or work plan) that outlines the analytic and interpretive approaches that will be used in the risk assessment. The general concern and approach articulated in the summary statement developed during scoping are given greater detail in a study-specific conceptual model. The model comprises both graphic illustrations (see Figure 3-2) and narrative descriptions that explicitly identify sources, contaminants of concern (stressors), exposure pathways, potential receptors, and adverse human health effects that the risk assessment is going to evaluate.

FIGURE 3-2

Illustration of the scope of a risk assessment, indicating both pathways considered (bold lines) and pathways not considered. Source: EPA 2004a.

The review of the conceptual model led to significant savings in the application of the model for calculating air dispersion, exposure and risk estimation. More than a third of the possible analyses were shown to be unnecessary to address the problem formulated in the planning and scoping discussion [EPA 2002, p. E-6].

For important risk assessments, particularly controversial or precedent-setting ones, it may be advisable that the scientific and technical credibility of the conceptual model be examined with a peer-review process. Although the conceptual model serves as a guide for determining what types, amount, and quality of data are needed for the assessment to address the issues and concerns of interest to decision-makers, the analysis plan matches each element of the conceptual model with the analytic approach that the assessors initially intend to use to develop data or otherwise represent that element. Box 3-4 lists some of the major elements of an analysis plan (from EPA 2004a).

BOX 3-4

Major Elements of an Analysis Plan. In addressing the above aspects of the analysis, the plan should also clearly describe the following: How will quality be ensured in each step (e.g., what will be included in the quality assurance/quality control plans)? (more...)

Recognition of the Need to Strengthen the Use of Formative Design Stages

The specific nature and needs of the decision environment are often neglected in risk assessment, if there is no systematic approach (Crawford-Brown 1999). It is increasingly clear that even “the highest-quality risk assessment is worthless if it does not address the needs of the decision-maker” (Suter 2006, p. 4). EPA guidance documents make it evident that the agency recognizes, at least in theory, that “[planning and scoping] may be the most important step in the risk assessment process” and that “without adequate [planning and scoping], most risk assessments will not succeed in providing the type of information that risk management needs to make a well founded decision” (EPA 2004a, p. 5-9). Similar ideas were also expressed in a report on EPA risk-assessment practices by GAO (2006). EPA has also observed that many of the shortcomings or failures of ecologic risk assessments can be traced to a weakness in or lack of problem formulation (CENR 1999).

Both the planning and scoping and problem-formulation stages are necessary to ensure that the form and content of a risk assessment are determined by the nature of the decision to be supported. Both stages offer opportunities to reach some level of consensus on how to proceed (for example, with respect to regulatory context and objectives, scientific objectives, data needs, or reasonably expected limitations) in an assessment so that its results will be useful and informative to decision-makers. Those stages also offer excellent opportunities to give risk communication an early and pivotal role in the overall risk-assessment process rather than allowing it to become an afterthought. Although both planning and scoping and problem formulation can be challenging and time-consuming, the time and effort are usually well spent and have been shown to result in risk assessments that are more useful to and better accepted by decision-makers (EPA 2002, 2003, 2004a).

The incorporation of those stages in the risk-assessment process is, however, still inconsistent. For example, both stages are missing from EPA’s new cancer guidelines and from the current Office of Solid Waste and Emergency Response risk-assessment protocol for combustion facilities (EPA 2005a,b). Thus, although the stages are now widely acknowledged, at least conceptually, as critical for the success of a risk assessment (particularly for complex, controversial, or precedent-setting assessments) and guidance for their conduct is available, the question remains as to whether EPA or other public agencies, the regulated community, or their contractors are taking full advantage of them to focus, refine, and improve human health and ecologic risk-assessment efforts. The question warrants attention in that continued inattention to the importance of planning and scoping and of problem formulation can be expected to yield human health risk assessments (by EPA and others) that fail to reach their full potential in providing support to decision-makers and others seeking solutions to environmental and health concerns.

Scylla and Charybdis: ² Navigating the Twin Hazards of Uncertainty and Delay in Decision Support

The combination of the magnitude and the practical irreducibility of key uncertainties and their impact on decision-making constitute the core challenge in efforts to achieve a robust yet practical approach to public-health and environmental risk management. The combination of uncertainty and the prospect of delaying important decisions constitute a key hazard in navigating the difficult waters of health and environmental decision-making. To an extent, it can be argued that the conflict is inherent in a decision-making environment that, while valuing a timely decision, places a large premium on the often-repeated yet ill-defined goal that the decision be “scientific,” “based on sound science,” or “based on the best available science.” The nature of the conflict can be understood by recalling that the scientific process of seeking the truth, by design and to its great credit, has no natural end point. Also by design, the training of scientists and such embedded traditions as applying tests of statistical significance instill the value of prudence and the “due-process” tasks of peer review, replication, and scientific debate before a conclusion can be said to be based in science. The idea that there are risks (for example, prolonged exposure to a hazard, or stress in the community awaiting an assessment of health risks) that may be associated with waiting for a particular study to be completed or for a scientific consensus to emerge is not readily incorporated into the standard scientific paradigm.

The lack of established “stopping criteria” in science contributes to the conflict wherein any attempt to put an end to or otherwise constrain scientific inquiry and debate to meet regulatory or legal deadlines or, perhaps most problematically, to achieve an abstract notion of timeliness can lead to the accusation that the corresponding decision is “unscientific.” In pursuing the goal of timely decision-making, there is an inherent conflict between meeting the requirements associated with the goal of knowing and the requirements associated with the more pragmatic goal of deciding.

Protection of the public and protection of the scientific knowledge base from Type 1 errors (that is, avoiding false positives) are not equivalent goals. That fact, somewhat obvious when considered carefully, is a fundamental source of tension that is not sufficiently acknowledged or confronted directly in risk assessment, risk communication, and risk management practices. Navigating (as opposed to resolving) the conflict between those goals is best addressed through its careful consideration by both risk managers and risk assessors in the formative and iterative design of risk assessment. To confront that challenge, risk managers must see themselves as managing uncertainty and delay as well as managing risk. Managing under uncertainty requires diverse strategies that address different aspects of the overall decision-making process, including investments to collect, store, and manage information; investments to improve the knowledge base, that is, to generate new knowledge; formalization of the processes used to collect, use, and process information; formalization of processes to calculate and communicate uncertainty; adjustment of the risk-assessment process to mitigate the practical impact of the uncertainty on the analytic process; adjustment of the decision-making process to accommodate the consideration of the uncertainty; and adjustment of the timing of decision-making in both directions—to delay or to expedite—when uncertainty is acknowledged to be sufficiently great.

It is important to note that the day-to-day work of uncertainty management should not be considered the sole domain of analytic experts. It is primarily the responsibility of the risk manager to prescribe and implement appropriate accommodations in the overall decision-making process if the analytic efforts aimed at supporting decisions under great uncertainty are to have the desired impact and to ensure that risks associated with delays in decision-making are balanced by the likelihood and magnitude of any benefit that is believed to be associated with proposed enhancements of the knowledge base or with the process of risk assessment. Choosing a strategy involves important tradeoffs because any strategy to deal with uncertainty will be incomplete and imperfect. The committee believes that one of the dominant pathways to improving risk analysis involves correctly matching the uncertainty-management strategy to the particular demands and resources of the decision-making envi ronments in and outside EPA. This issue is discussed in greater detail in Chapter 4. Ideally, the matching process would be expanded to consider the many other decision-makers that make use of EPA’s analytic products.

Value of Information: What Makes Information Valuable?

A fundamental aspect of decision-making under uncertainty involves the inevitable choice between making an immediate decision with the information and analysis available and delaying the decision while, for example, more raw information is collected, a more refined analytic product is prepared, or consultations with affected parties are conducted. Even if delay is not the primary concern, the direct and indirect costs of acquiring the information will often need to be considered.

As the most generic analytic framework for valuing information in the context of decisions, value-of-information (VOI) analysis provides a set of methods for optimizing efforts and resources to gather, to process, and to apply information to help decision-makers achieve their objectives. The application of VOI analysis is illustrated schematically in Figure 3-3.

FIGURE 3-3

Schematic of the application of value-of-information analysis to assess the impacts of additional studies in a specific decision context. Information opportunities that address uncertainties in the baseline model are considered with respect to the changes (more...)

The Process of Quantitative Value-of-Information Analysis

The decision-theoretic process to quantitatively value information begins with analyzing the best option available to the decision-maker in a certain state of uncertainty. This serves as a baseline scenario with respect to information available to the decision-maker. The process then systematically considers when and how the decision-maker’s preferred option might be changed if the decision-maker was able to incorporate additional information into the decision that was not available in the baseline scenario. This new information is expected to either eliminate or reduce the extent of a source of uncertainty.

In VOI analysis, the decision-maker is assumed to change the preferred option only when there would be a change in the net expected benefits. Accordingly, in addition to consideration of how likely it is that the preferred decision would change, the process measures how much of an increase in benefit would be expected given the additional information. The net (or expected) value of gathering information to resolve or reduce uncertainty is calculated by weighing the increase in benefits associated with each potential outcome of the information collected by the probability of each outcome. This weighing process includes assigning the value zero (that is, representing no increase in benefits) for situations where the information gathered does not change the decision-maker’s preferred option.

A critical part of understanding the concept of VOI analysis is to differentiate scientific and decision-analytic perspectives on the value of information. In research proposals and in the literature, scientists often describe proposed studies as valuable with respect to enhancing the overall knowledge base, perhaps with a suggestion that it will inform important decisions. Conversely, the decision-analytic notion of VOI is entirely decision-centric. In a VOI analysis, an information source is valued solely on the basis of the probability and magnitude of its potential impacts on a specific decision at a specific time with a specific state of prior knowledge. Therefore, it is a common and expected result of VOI analysis to estimate that an information source, which may otherwise be considered valuable as a general scientific matter, has little or no value in support of a particular decision. This happens when the specific decision is not sensitive to the resolution of the uncertainty that the information source addresses. Considering this situation in Figure 3-3, the arrow indicator, which denotes that option C is preferred given currently available information, would not be moved much by this source of new information.

Experience in the Application of Value-of-Information Methods

The applications of VOI methods in environmental health decision-making might be characterized as sporadic and somewhat academic (Yokota and Thompson 2004). In the academic literature, there has been a considerable interest in the use of VOI techniques to evaluate various activities within toxicity testing (Lave and Omenn 1986; Lave et al. 1988; Taylor et al. 1993; Yokota et al. 2004). Recently Hattis and Lynch (2007) applied a VOI framework to assess the expected effect of improved human pharmacodynamic or pharmacokinetic variability information on doses deemed to be protective for noncancer effects. VOI methods have been employed to estimate value of sampling information in the context of environmental remediation (Dakins et al. 1996), and in an assessment of information value in the context of alternate control policies for source water protection in a watershed impacted by agricultural runoff (Borisova et al. 2005). Other applications can be found in the value of improved exposure information in the case of drycleaning operations (Thompson and Evans 1997), and the value of genetic screening options related to prevention of beryllium disease (Bartel et al. 2000).

There is evidence of sporadic interest and research aimed at employing VOI methods at EPA. For example, Messner and Murphy (2005) present an analysis of VOI about the quality of source water in the context of decisions about investments in drinking water treatment plants. In other applications, EPA staff and contractors have applied VOI principles in assessing the value of environmental information systems and human exposure information across a class of regulatory decisions (IEc 2000; Koines 2005).

Prospects for Formal Value-of-Information Analysis at EPA

VOI analysis has a number of benefits in support of decision-making compared with the more common scientific characterization of the potential value of a study. The intuitive and idiosyncratic views of individual scientists and decision-makers tend to place high value on information from their own discipline while diminishing the value of information from other disciplines. Scientists from all disciplines may devalue information that is not scientifically interesting (for example, that would not be publishable in a scientific journal) even if it substantially reduces a critical uncertainty in a risk assessment and the knowledge has considerable potential to affect the decision-maker’s choice of the best option. In contrast, VOI analysis could provide both a more context-specific and a more objective assessment of the decision-centric value of a piece of information or, by extension, the value of an information system to a class of decisions that might use it. Despite the potential benefits, it is important to note that a VOI analysis is not considered to be generally superior to the use of expert scientific judgment about the importance of a scientific investigation; rather, it answers a much narrower question about the importance of a study for the outcome of a specific decision and is not appropriate as a general measure of the scientific merit and broader utility of a study.

For example, in the context of some specific decision, a VOI analysis might place great value on a small survey to estimate the fraction of businesses using a near-obsolete technology and very little value on a large, well-designed, and broadly important scientific study when considering only the narrow purposes of the specific decision at hand. The decision-maker’s preferences for options (perhaps in choosing among options B, C, and D in Figure 3-3) may be very sensitive to the level of uncertainty in risk reductions and the costs that would be imposed on businesses by a decision that would, for example, forbid the continued use of the older technology. In both the risk estimation and the cost estimation, the number of such businesses may be an important consideration in this particular decision context. Conversely, a scientific study that would contribute to the understanding of the risk and may reduce the overall uncertainty in a broadly desirable and scientifically rigorous way may not be able to add information that changes the relative desirability of the specific options enough to change the decision-maker’s preferred choice. Clearly, there are many other scenarios in which scientific investigation is precisely what is required to differentiate adequately among available options.

Despite the intellectual appeal of the formal VOI analytic framework and the ever-present need for a robust means of assessing information value, the formal VOI paradigm imposes a number of challenges that limit its practical and widespread use in the near term. The use of the formal VOI framework in environmental health applications has been extensively reviewed by Yokota and Thompson (2004). One of their findings relates to the somewhat academic status of VOI in this field:

Rigorous VOI analyses provide opportunities to evaluate strategies to collect information to improve EHRM [environmental health risk-management] decisions. This review of the methodology and applications shows that advances in computing tools allow analysts to tackle problems with greater complexity, although the literature still lacks “real” applications, probably due to a number of barriers. These barriers include the lack of guidance from EPA and others on criteria for standardizing EHRM risk and decision analyses, the lack of consensus on values to use for health outcomes, the lack of default distributions for frequently used inputs, and inexperience of risk managers and communicators with using probabilistic risk results.

There are important considerations in addition to the barriers expressed above.

• VOI computation can be technically challenging, particularly when one is trying to evaluate imperfect information, which is almost always the relevant case.
• Its analytic formality does not lend itself to being combined with the more common deliberative approaches of determining the potential value of information.
• The approach presumes that the analyst can fully describe the change in a decision-maker’s choices in response to new information. This condition is not very realistic (or at least is rarely the case) and is particularly problematic when the decision-making process is not rule-driven or whenever the VOI analyst is forced to speculate as to the behavior of the decision-maker in response to new information.
• The impact of the new information must be characterized with respect to the resulting change in a probability distribution that describes the current level of uncertainty, which may not be formally characterized as a probability distribution.
• Very few technical or policy analysts or decision-makers have had any exposure to this type of analysis, suggesting a considerable burden of training.
• The “value” assigned in a VOI analysis is itself, ultimately, an uncertain quantity.

A key challenge for uncertainty management in EPA and elsewhere is the need to design the risk assessment to support decisions with respect to an explicit array of candidate options that the decision-maker is likely to consider. Without these options, it is not possible to assert a formal decision-centric valuation of information; indeed, in this case, a formal VOI analysis cannot even be attempted. A key potential side effect is the perpetuation of “incomplete” risk assessments. The perpetuation side effect is a natural result in the absence of a well-characterized decision-support context, including a concrete array of decision options, because there will always be a scientific rationale, as opposed to a decision-centric rationale, to continue to gather information, perform or review new studies, and to improve technical aspects of a risk assessment.

The committee recognizes both the advantages of VOI analysis for risk assessment and risk management as well as the presence of continuing barriers to the use of formal and computational VOI analysis in EPA. As a result, there is likely to be only a small proportion of risk assessments and decision contexts that meet the criteria where a formal VOI is possible (for example, having clear decision rules and prior estimates of uncertainty) and for which the stakes are high enough to make a VOI analysis cost-effective.

Alternative VOI Methods for Diverse Decision-Making Contexts

As an alternative that is applicable to a larger proportion of decision contexts, the committee believes that EPA would benefit from developing and applying a structured but less quantitative method for assessing the value of new information that captures the essential reasoning embodied in VOI analysis. The essential reasoning in the formal VOI approach is based on the explicit characterization of a direct causal link between a specific source of new information, the predicted change in the behavior of a decision-maker given this new information, and the resulting improvement with respect to the decision-maker’s objectives that can be expected in the presence of the additional knowledge. Essentially, the process of valuation would involve the presentation of a qualitative or semiquantitative argument (as opposed to formal computation) that describes the causal relationship between the knowledge that might come from the considered source of information and the potential for improved decision outcomes. The process could also consider the potential for risk in delaying the decision until the information is available and is adequately incorporated into the decision-support products (either risk assessments or cost assessments). An example of the development and application of a structured semiquantitative VOI method, including a discussion of the complementary role of these methods, can be found in Hammitt and Cave (1991).

Valuing Methodologic and Procedural Improvements in Risk-Assessment Design

Earlier in this chapter, the committee described the rationale for placing a great premium on aspects of process in risk assessment. When all the combinations of choices of scope and technical, consultative, and quality-control methods are considered with the variations in the intensity of their application, it could be argued that there are an uncountable number of ways in which a risk assessment could be constructed. Such flexibility is generally welcome and has the potential to make risk assessment relevant to the broadest possible array of applications, but it can be problematic.

The essentially deliberative process of matching opportunities to enhance the risk-assessment process with the objectives of achieving high-quality decision support may be facilitated by using a decision-centric evaluation model that characterizes the impact of any proposed enhancements to the risk assessment—and its manifestation in the form of a risk-assessment product with corresponding attributes—on the desired objectives of the decision-making function. The committee encourages the development of such an evaluation framework for methodologic improvements in risk assessment that instills some of the concepts of decision-analytic value of information. A schematic of such an evaluation model is illustrated in Figure 3-4.

FIGURE 3-4

Schematic of an analysis of the value of various methodologic opportunities (or “value of methods” analysis) to enhance the risk-assessment process and products. The structure mimics the standard VOI approach, but focuses on different (more...)

The proposed evaluation framework would expand the consideration of the casual relationship between risk-assessment activity and the quality of decision-making in two respects. It would be structured to assist in the relative valuation of the many attributes of risk-assessment processes and products that need to be considered in the formative and iterative design process. By relaxing the formality of the VOI approach, it could include a broader set of decision-making objectives—such as transparency, timeliness, integration with other decision inputs, and compatibility with stakeholder participation—that are less tangible and quantifiable but nonetheless critically important in determining the overall decision-support value of a given activity or effort.

An important aspect of instilling the benefits that are analogous to VOI analysis will be in drawing explicit causal linkages, even if expressed qualitatively, between risk-assessment design options and the ultimate impact on the decision-making environment. In this way, the potential for the “value-of-methods” approach is limited in an analogous way by one of the barriers in the formal VOI approach. In VOI analysis, the analyst must know the decision-maker’s valuation of risk assessment or other quantitative outcomes in sufficient detail as to predict a change in the decision-maker’s behavior in response to new information (that is, predicting their choice among available options, or their choice in setting a single number within a continuum). In the value-of-methods approach, the analyst who is contemplating the value of a particular risk-assessment method (for example, in choosing among a qualitative, quantitative scenario-based, or fully probabilistic characterization of uncertainty) requires some way to characterize the change in the decision-support environment that corresponds to each of these alternative methods. Further, the analyst would need to know how much the different changes in the decision support environment are valued based on the capacity of the decision-making process to take advantage of the method, and the institutional values of the desirable qualities of decision-making. In order to remove this potential barrier, this expression of the valued attributes of decision support would be made highly context specific (for example, having very different objectives for community-level decision support as compared to a national standard-setting process) and would be agreed to and documented in the formative stages of risk-assessment design.

Weight-of-Evidence and Hazard Classification: An Example of a Value-of-Methods Question

The phrase weight of evidence (WOE) is used by EPA and other scientific bodies to describe the strength of the scientific inferences that can be drawn from a given body of evidence. In its most common applications in EPA, WOE is used to characterize the hazardous (toxic or carcinogenic) properties of chemicals on the basis of an integrated analysis of all relevant observational and experimental data. It is increasingly used to describe the strength of evidence supporting particular modes of (toxic) action (MOAs) and dose-response relationships. Because scientific evidence used in WOE evaluations varies greatly among chemicals and other hazardous agents in type, quantity, and quality, it is not possible to describe the WOE evaluation in other than relatively general terms. It is thus not unexpected that WOE judgments in particular cases can vary among experts and that consensus is sometimes difficult to achieve.

Perhaps the most formal WOE activity undertaken in EPA concerns the classification of carcinogens. The weighing of evidence from epidemiology and experimental studies pertaining to specific chemicals or chemical mixtures that may be carcinogenic involves substantial agency resources and can lead to controversy and extended debate.

One distinction made in EPA carcinogen classification is whether the available evidence is sufficient to establish causality for humans (that is, whether a substance can be labeled as a “known” human carcinogen) or falls short and indicates that the agent is a “likely” human carcinogen. Causal relationships can be more straightforward to establish in well-done clinical and (in animals) experimental studies, but an individual observational (epidemiology) study typically can establish only a statistical association. A larger body of epidemiologic evidence can be sufficient to rule out bias and confounding with sufficient confidence to support a causal relationship; with experimental evidence, it may be sufficient to establish causality in humans. The weighing of such evidence can be controversial, so such institutions as EPA, the International Agency for Research on Cancer (IARC), the Institute of Medicine (IOM), and the National Toxicology Program (NTP) have developed practices and classification schemes to aid the process of reaching conclusions about the overall evidence. NTP, IOM, and IARC convene expert bodies to undertake WOE analyses of carcinogenicity data; EPA relies on peer review by expert groups, such as its Science Advisory Board, to vet staff findings on carcinogenicity evidence.

The committee notes that in some cases there does not appear to be substantial value in the agency’s making distinctions between certain carcinogenicity classifications. Whether a chemical is “carcinogenic in humans” or “likely to be carcinogenic in humans” generally has no important influence on the ultimate quantification of risk and the use of risk estimates in decision-making. In many regulatory contexts, known human carcinogens may be treated no differently from “likely” human carcinogens: risks are estimated for all substances for which there is sufficiently convincing evidence of carcinogenicity, irrespective of whether human causality has been established, and the risk estimates are not adjusted according to the WOE classification.

As a result, once the available evidence, either epidemiologic or experimental, is judged sufficient to establish that a given finding of toxicity or carcinogenicity is potentially relevant to humans, there may not be the need for further distinctions in classification, except in some circumstances as a communication tool. Unless clear reasons are brought forward at some stage, such as in the formative design stage of risk assessment, to support the need for such a definitive human causality assessment, the committee sees no reason for the agency to spend time and resources to fine-tune the hazard classification in order to settle the question of whether the agent is a likely or known cause of the effect in humans.

However, the systematic consideration of evidence in WOE analyses remains important as a matter of good scientific practice. Thus, whether the accumulated evidence is sufficient to consider a substance potentially hazardous to humans or is sufficient to support a given MOA requires a weighing of individual studies and pieces of evidence, and this practice should continue. The committee recommends that the agency remain mindful of cases in which fine distinctions have little or no impact on the overall use of risk information.

WOE classification provides an example of distinctions between the formal VOI analysis and the less formal value-of-methods analysis. The fact that these finer distinctions in WOE classification are not used further in risk assessment or in any apparent decision rule used by EPA suggests placing no value on the exercise to seek these distinctions, when the potential benefit is viewed purely from a formal VOI analysis perspective (as illustrated in Figure 3-3). But a WOE classification that distinguishes known from likely carcinogens may be deemed by EPA to be required in support of other values associated with risk assessment practice (for example, using a “good scientific practices” argument, or as the basis for a simplified means of communication of the epistemic status of a claim of carcinogenicity). WOE is an example of how EPA may benefit from a structured characterization (as described above and illustrated in Figure 3-4) of the exact role of a resource-intensive method in supporting the broader goals of public-health and environmental decision-making, which would include, among many other aspects, the use of good scientific practices and consideration of good communication practices. The method would require a more explicit valuation of important attributes of quality in decision support.