Types of Evidence

Many types of evidence can be considered when looking at the relationship between exposure to a toxic substance and health effects or disease. This evidence can include large, well-conducted clinical controlled trials such as those used for pharmaceutical agents; epidemiologic studies where groups of humans are evaluated to see if exposures have an impact on health; animal studies to determine the toxicity of an agent; and mechanistic studies, often conducted in vitro or at the subcellular level to determine the biological mechanisms by which an agent produces an outcome. Each of these studies is described briefly below to highlight the wealth of information that may be considered when evaluating the impact of an agent on the health of an organism. The committee notes that this section is not intended to be a comprehensive description of epidemiology, toxicology, or occupational medicine.

Approaches for Establishing Causal Links

In 1965, following the U.S. surgeon general's report on the relationship between smoking and lung cancer, Sir Austin Bradford Hill, a British epidemiologist and statistician, described nine viewpoints (often referred to as criteria) that should be considered when trying to come to a decision about whether an observed association might be causal (Hill, 1965). While all viewpoints are relevant in making inference about causality there is only one of the nine viewpoints that is truly necessary—temporality—that is, the exposure must have occurred before the onset of the disease. The remaining eight viewpoints are neither necessary nor sufficient requirements for causation, but nonetheless provide a framework for consideration (see Box 2-2).

BOX 2-2

Hill's Viewpoints. Strength: Causation is supported if the relative risk due to the exposure is very large. Consistency: Causation is supported if the relationship is seen in different populations at different times and in different circumstances.

There are various approaches in addition to Hill's to evaluate causal associations between exposure to an agent and disease, most of which emphasize the need for a strong association and a defined biological mechanism based on a combination of human, animal, and mechanistic data (EPA, 2012). Organizations such as the NTP, the EPA, and IARC periodically conduct evaluations of agents that are suspected causes of disease. Such evaluations are typically peer-reviewed by a panel of experts who have been selected after consideration of scientific background and potential conflicts of interest, and for federal groups such as advisory committees, of bias. The resulting evaluations consider the available evidence derived from human and animal experiments, epidemiological research, and basic mechanistic studies.

The approaches used by the aforementioned organizations to evaluate causality by a group of experts all use a weight-of-evidence approach to characterize the degree of uncertainty in the evidence base. This uncertainty is frequently expressed by categories describing the strength of the evidence base irrespective of the strength of the association between exposure and health effect. Given the variable quantity and quality of evidence for many toxic substances, it is helpful to have a panel of experts review the evidence and come to a consensus on the conclusion. The breadth and depth of knowledge gained from using a number of experts gives additional support to conclusions and can help minimize bias and error. The committee discusses the approaches used by IARC, NTP, and the IOM as representative of organizations that have earned respect for high-quality reviews of scientific evidence regarding potential causal associations between substances and health effects.

International Agency for Research on Cancer

The approach used by IARC to evaluate whether an agent is a carcinogen includes consideration of human, animal, and mechanistic evidence (see Figure 2-2). The methods and criteria IARC uses to evaluate the various kinds of evidence are clearly described (IARC, 2006). In evaluating human evidence, for example, after the quality of individual epidemiologic studies of cancer has been summarized and assessed, a judgment is made concerning the strength of evidence that the agent in question is carcinogenic in humans. In making its judgment, each IARC working group considers several criteria for causality (e.g., the Hill viewpoints) (Hill, 1965). A strong association——for example, a large relative risk, with a confidence interval that does not include 1.0, representing a greater probability of the disease in the exposed group versus that in a nonexposed group—is more likely to indicate causality than a weak association, although it is recognized that estimates of a small effect do not imply lack of causality and may be important if the disease or exposure is common. Associations that are replicated in several studies of the same design or that are observed using different epidemiologic approaches or under different exposure scenarios are more likely to represent a causal relationship than are isolated observations from single studies. If there are inconsistent results among investigations, possible explanations are considered (such as differences in exposure), and results of studies that are judged to be of high quality are given more weight than those of studies that are judged to be methodologically less sound (IARC, 2006).

FIGURE 2-2

IARC approach to evaluating the evidence to determine the carcinogenicity of hazardous agents. NOTE: ESLC = evidence suggests lack of carcinogenicity. Arrows indicate a potential change in a substance's carcinogenicity group based on the strength of evidence (more...)

In evaluations of carcinogens published in the IARC monograph series, the evidence of cancer in humans and in experimental animals has four descriptors—“sufficient evidence,” “limited evidence,” “inadequate evidence,” or “evidence suggesting lack of carcinogenicity” (for definitions of these terms, see IARC, 2006). These evaluations regarding human and animal evidence are combined into an evaluation that the agent is

• carcinogenic to humans (Group 1),
• probably carcinogenic to humans (Group 2A),
• possibly carcinogenic to humans (Group 2B),
• not classifiable as to its carcinogenicity to humans (Group 3), or
• probably not carcinogenic to humans (Group 4) (see Figure 2-2).

When a judgment has been reached that there is sufficient evidence that an agent causes disease, it means that a working group of subject matter experts has evaluated and judged the body of evidence and concluded that a relationship has been observed between the exposure and cancer in studies in which chance, bias, and confounding could be ruled out with reasonable confidence (IARC, 2006).

Institute of Medicine

Many IOM committees use a formal system similar to those of IARC and NTP to assess the weight-of-evidence and determine the strength of association between exposure to a hazardous agent and a health outcome (IOM, 2010). As with other approaches to assessing cause and effect, the IOM uses committees of experts who first evaluate the available evidence (typically from peer-reviewed studies). All committee members then reach a consensus on the conclusion and assign a category of association for each health outcome based on the number and quality of studies and expert judgment (see Box 2-3). The committees do not use a formulaic approach to assign a specific category of association; rather they have found that each health outcome requires a more considered and nuanced approach (IOM, 2010). EPA has used the IOM approach for its integrated scientific assessments for criteria air pollutants since 2008.

BOX 2-3

Categories of Associations Used by Several IOM Committees. Sufficient Evidence of a Causal Relationship: Evidence is sufficient to conclude that a causal relationship exists between an exposure and a health outcome. The evidence fulfills the criteria (more...)

Establishing Hazardous Agent—Occupational Disease Links in Haz-Map

The committee's understanding of the Haz-Map database approach to linking hazardous agents and occupational diseases is based on information provided by its developer and found at www.haz-map.com. The committee notes that this information is not provided on the NLM Haz-Map website—located at http://hazmap.nlm.nih.gov—although there is a link for direct users to more information at www.haz-map.com. The developer told the committee that he tries to answer the following questions when determining whether exposure to a hazardous agent may cause an occupational disease: “Is there consensus in occupational medicine textbooks that this occupational disease is caused by these hazardous agents? Can the disease be prevented by good occupational hygiene practices?” (Brown, 2012d).

The Haz-Map developer has determined that

linkage between a chemical or biological agent and a disease indicates that sufficient exposure to the agent is associated with increased risk of developing the disease. For chronic diseases, links between an agent and a disease means that a causal relationship has been determined based on human case reports or epidemiological studies. (Brown, 2012b)

He further states that

In Haz-Map, there is a distinction between adverse effects (includes animal toxicology and human poisonings by ingestion cases) and occupational diseases (cases of workers made ill after inhalation or skin absorption). Therefore, chemicals are linked to the diseases “Asphyxiation, chemical” and “Hemolytic anemia” only if occupational cases (and not just ingestion cases) have been reported. Likewise, all chronic occupational diseases in Haz-Map are based on reports of occupational cases. (Brown, 2012d)

The www.haz-map.com website provides more information on some of the hazardous agent—occupational disease links in the database:

The only exceptions to the rule that linkage indicates human disease known to be caused by the agent are the following diseases: Solvents, acute toxic effect (linked to all organic solvents); Encephalopathy, chronic solvent (linked to all organic solvents used in paints or varnishes); Encephalopathy, acute toxic effect (linked to other potential causes of acute encephalopathy excluding solvents, asphyxiants, fumigants, and insecticides); and Pneumonitis, toxic (caused by chemicals corrosive to the skin, eyes, and respiratory tract). The occupational disease “pneumonitis, toxic” is defined as noncardiogenic pulmonary edema induced by acute exposure to metal fumes or toxic gases and vapors after a spill or confined space accident. The “major irritant inhalants” are ammonia, bromine, chlorine, chlorine dioxide, diborane, ethylene oxide, formaldehyde, hydrogen chloride, hydrogen fluoride, methyl isocyanate, nitrogen dioxide, ozone, phosgene, and sulfur dioxide [LaDou, p. 523]. While human cases have been documented for the major irritant inhalants, there are many chemicals with similar properties that can cause acute lung injury. Haz-Map flags 560 chemicals that have the potential to cause toxic pneumonitis as an adverse effect in heavily exposed workers or in animal experiments. Of these 560 chemicals, 142 with the designation of “toxic inhalation hazard” by ERG 2008 are linked to the occupational disease “Pneumonitis, toxic.” (Brown, 2012b)

More specifics on what constitutes a consensus, what evidence is reviewed in making the links, and how the Haz-Map developer determines that good industrial hygiene practices would prevent the disease are not provided on the database website.

Toxic substance—disease links are presented in the field “Diseases” under the category “Related Information in Haz-Map” (see Table 2-1 and Box 2-1). The determination of what diseases are associated with the agent is based on a review of the literature by the developer, although much of the reviewed literature is of secondary sources rather than primary studies. Furthermore, the developer does not indicate the criteria he uses to determine “sufficient exposure,” “increased risk,” or a “causal relationship.” The evidence used to determine associations or causal links is not provided for specific agents or generally for the database, so it is unclear how the developer made each determination and how robust the agent—disease link actually is. For example, it is unknown if the basis of a particular toxic substance—disease link is one well-conducted epidemiologic study, five adequate epidemiologic studies, three case studies, conflicting studies, or a lack of studies. For noncancer health endpoints, however, there is no evidence that the developer uses a weight-of-evidence approach for determining causality, such as that used by NTP or IOM.

The exception to this lack of specificity are those agents that are IARC Group 1 carcinogens (sufficient evidence in humans) as described in the chapter on occupation by Siemiatycki and colleagues in Schottenfeld and Fraumeni's Cancer Epidemiology and Prevention, 3rd ed. (Brown, 2010; Schottenfeld and Fraumeni, 2006; Siemiatycki et al., 2006). This Haz-Map approach has recently been updated. Target site organs are determined by using IARC 2012 cancer site information. According to the developer,

Prior to the 2012 IARC changes, the map of occupational cancer in Haz-Map was based on the “Occupation” chapter in Cancer Epidemiology and Prevention, 3rd Edition. New studies and new interpretations from IARC are now available. (Brown, 2012a)

Selection and Interpretation of Information Sources

There are several areas where Haz-Map lacks transparency. The most critical is the lack of formal criteria for determining the hazardous agent—occupational disease links. However, even before such criteria could be applied to the evidence selected by the developer, there is a lack of transparency about what information is reviewed, its sources, and how it is evaluated for each hazardous agent, with the exception of IARC classifications for carcinogens. For example, there are four noncancer diseases causally associated with formaldehyde—asthma, occupational; contact dermatitis, allergic; contact urticaria; and fumigants, acute toxic effect. The sources of the information for contact urticaria and fumigants, acute toxic effect are not given in the “Diseases” field, but rather must be deciphered from the earlier “Comment” field in the “Agent information” category. Further references can also be found by clicking on the specific disease link and reading the entry in the “Comment” field for that specific disease. Although sources for the information on asthma, contact urticaria, and allergic contact dermatitis are provided, the casual user would not easily find the supporting evidence for these agent—disease links in the database.

Criteria for Determining Agent—Disease Links

The rules and criteria used by the Haz-Map developer to determine whether the evidence is sufficient for a causal association between an agent and an occupational disease are not clear, with the exception of the very strict criteria of using only IARC Group 1 cancer designations. Furthermore, there appear to be discrepancies in using even this approach for carcinogens. Although IARC has designated asbestos as a known human carcinogen for ovarian cancer, this is not indicated in the asbestos record. However, cancer of the larynx, which was designated by IARC as caused by asbestos at the same time as ovarian cancer, is listed in the database as an asbestos-caused occupational disease.

For chronic noncancer occupational diseases, the database appears to favor evidence from epidemiological studies and reports of occupational illness. However, in the absence of human information, it does not consider other types of information (e.g., animal and mechanistic data) that may support a link between an agent and occupational disease. For acute occupational diseases, the developer states that

animal data is sufficient if the routes of entry correspond. Examples of such acute occupational diseases include poisoning by pesticides, solvents, simple asphyxiation, hydrofluoric acid, and toxic pneumonitis. A special rule is applied to toxic pneumonitis. Any corrosive substance has the potential to produce toxic pneumonitis as an adverse effect. Any of these substances designated as “TIH” (Toxic Inhalation Hazards) in the 2008 Emergency Response Guidelines are also listed in Haz-Map as the occupational disease “Pneumonitis, toxic.” Other acute diseases with special rules are “Asphyxiation, chemical” and “Hemolytic anemia.” In Haz-Map, there is a distinction between adverse effects (includes animal toxicology and human poisonings by ingestion cases) and occupational diseases (cases of workers made ill after inhalation or skin absorption). Therefore, chemicals are linked to the diseases “Asphyxiation, chemical” and “Hemolytic anemia” only if occupational cases (and not just ingestion cases) have been reported. Likewise, all chronic occupational diseases in Haz-Map are based on reports of occupational cases. (Brown, 2012d)

The stringent criteria for establishing hazardous agent and cancer links and the uninterpretable criteria for establishing noncancer disease links present substantial obstacles for the effective use of Haz-Map as the sole source of toxic substance—occupational disease links in SEM as discussed in detail in Chapter 3.

Peer Review Concerns for Haz-Map

Peer review is a commonly used process for scientific and technical articles submitted to scholarly journals. The goal of peer review is to help ensure that the reviewed documents are accurate, comprehensive, and, in some cases, adhere to the authoring organization's policy guidelines. Peer review has been defined by the National Research Council as

a documented, critical review performed by peers [defined in the U.S. Nuclear Regulatory Commission report as “a person having technical expertise in the subject matter to be reviewed (or a subset of the subject matter to be reviewed) to a degree at least equivalent to that needed for the original work”] who are independent of the work being reviewed. The peer's independence from the work being reviewed means that the peer, a) was not involved as a participant, supervisor, technical reviewer, or advisor in the work being reviewed, and b) to the extent practical, has sufficient freedom from funding considerations to assure the work is impartially reviewed. (NRC, 1997)

Typically, organizations have established criteria against which reviewers are asked to judge the document. Numerous organizations, including virtually all biomedical journals (e.g., Journal of the American Medical Association, the Lancet, New England Journal of Medicine), and other organizations such as the IOM and the National Research Council, NTP, ATSDR, EPA, IARC, and the Organisation for Economic Co-operation and Development, use peer review to ensure their documents meet scientific standards. Many government and other organizations have developed and documented their peer review process. Some organizations, such as NTP and IARC, publish their review guidelines so the public has an understanding of the level of rigor the organization has applied to its review process. For example, EPA has developed the Peer Review Handbook that provides guidance to EPA staff on how and when to conduct peer reviews of scientific documents and the types of peer review that are applicable to different documents (EPA, 2006). The Office of Management and Budget issued its “Final Information Quality Bulletin for Peer Review” that established federal guidance for the peer review of government science documents (OMB, 2004). The peer reviewers' names are identified in many documents, although their reviews may be anonymous to the documents' authors during the review process.

Although the NLM publishes the database, Haz-Map lacks adequate oversight or content review by external, independent experts because the Haz-Map developer is solely responsible for its content. Furthermore, there is no disclaimer on the Haz-Map home page at NLM indicating that it is not peer-reviewed or that NLM is not responsible for its content. Thus, the user may be unaware that this database has not been peer-reviewed nor has it been reviewed by anyone at NLM for accuracy, bias, or comprehensiveness. In contrast, NLM's HSDB prominently displays a notice on its home page that the database has been peer-reviewed (NLM, 2011).

The current Haz-Map process for determining toxic substance—occupational disease links is based on the developer identifying “textbooks, journal articles, the Documentation of the Threshold Limit Values (published by ACGIH), and electronic databases such as NLM's Hazardous Substances Data Bank (HSDB).” The developer then “classifies, summarizes, and regularly updates the information found in the database” (http://hazmap.nlm.nih.gov/about-us; accessed January 2, 2013). The developer of Haz-Map extracts information from these sources and uses his own expertise and judgment to determine whether there is sufficient evidence from his sources to determine whether there is a causal link between a toxic substance and an occupational disease. The substance—disease links receive no further review for accuracy or comprehensiveness.

Ideally, appropriate review of a database such as Haz-Map would include both a technical review and a quality assurance component. A technical review would ensure that the correct information from each source was cited accurately. The quality assurance component would determine that all fields are complete, that the cited information is not taken out of context, and that all the relevant information is included. The failure to list asbestos as a cause of ovarian cancer is one of the examples that the committee was able to identify during the course of the study that show why technical and quality control reviews are needed. The reason for excluding this cancer from the asbestos record is unclear. With good technical review and quality assurance of the database entries, users can make informed judgments about the validity of the links between a toxic substance and an occupational disease.

Haz-Map relies heavily on textbooks and standardized reference books to determine agent—disease link. These books include Cancer Epidemiology and Prevention (Schottenfeld and Fraumei, 2006), the source of the database's references to Siemiatycki et al. (2006) reference in Haz-Map; Textbook of Clinical Occupational and Environmental Medicine (Rosenstock et al., 2004); and Contact and Occupational Dermatology (Marks et al., 1997). Databases such as HSBD and REPROTOX are also used. Haz-Map cites only IARC for cancer designations.

Although some of the referenced databases, such as HSDB, are peer-reviewed, this is not necessarily true for textbooks and other information sources, including some databases. Textbooks are typically written as educational tools and are not designed to be either all-encompassing or used for compensation purposes. While subject to the scrutiny of editors, they are not peer-reviewed to determine if the authors' conclusions about the literature on a subject are accurate, comprehensive, or objective. Furthermore, although the information in many textbooks and, thus, in Haz-Map is presumably based on information from peer-reviewed and published documents, textbook chapters typically represent the interpretation of a limited number of authors who select which studies to review, evaluate, and summarize. Different conclusions about a chemical agent may be reached by different authors, depending on their purpose and their resources. Many of the references used for Haz-Map are not easily accessible to the general public either in hard copy or electronically, which makes it difficult to check the information from them to complete a quality assurance and technical review.

Updating Toxic Substance—Disease Links

Haz-Map is updated quarterly on the NLM website with some agents selected preferentially for review and updating (Brown, 2012d). Formal update information is not provided in the NLM version of Haz-Map making it impossible to ascertain which agents may be considered for review and updating and whether any changes have been made to the “Diseases” field. The developer provides some information on database updates at http://www.haz-map.com/wotsnu.htm, but this information can be difficult to interpret, such as the entry for January 7, 2013.

Updates to hazardous agent records in Haz-Map occur when a new edition of a textbook or other reference is published with revised information on that agent. In 2010, 2,400 substances listed in HSDB were added to Haz-Map, with another 600 chemicals and 10 diseases added in 2011 (Brown, 2012c). Further revisions to Haz-Map were made in December 2012. Hazardous agents may also be added to the database at the instigation of DOL. The SEM contractor is required to “develop a list of toxic substances whose Haz-Map profiles are to be prioritized for updating by the DOL Project Medical Consultant,” subject to approval by the DOL (see Chapter 3). Although the committee was not informed as to the identity of the DOL project medical consultant, the DOL contractor stated that “funding is provided to Dr. Brown for research of toxic substances of interest to DOL” (Stalnaker, 2012). DOL refers substances on the SEM list to the Haz-Map developer for priority consideration. He also occasionally reviews proposed agent—disease links that have come to the attention of DEEOIC staff, often from EEOICP claimants or their representatives (Karoline Anders, DOL DEEOIC, personal communication, October 9, 2012).