Multiple Imputation for Missing Data

Missing data can arise from settings in which people are asked about sensitive personal data, when resource constraints limit the completeness of data collection, or when certain items of information are not routinely collected. When a given variable is essential for a particular evaluation, analyses only of complete cases introduces many threats to inference: (1) bias can be introduced because persons with missing data may be systematically different from those with complete data; (2) statistical power can be reduced when many cases are deleted from analyses due to missing data; (3) resources can be wasted—for example, when 95 percent of data are collected on someone, but due to the 5 percent missing data, the entire data block is left unused; and (4) ethical obligations to research subjects can be compromised when they have inconvenienced themselves under the assumption that they were doing this for biomedical or behavioral research, but the investigator discards their data due to missing variables.

Data on sensitive topics such as sexual risk behaviors or drug use may be limited by nonresponse bias or biases stemming from socially sensitive responding. These biases present a special challenge to the collection of data for surveillance and for epidemiologic research studies where sexual behaviors or drug use may be relevant (Fenton et al., 2001). Multiple imputation helps to circumvent the need to eliminate subjects with partially observed data imputing (predicting) values for missing variables. Such imputation requires a statistical model for the complete data (including the unobserved portions) and for the process that led to the observed pattern of missing observations. This model is used to predict the missing observation based on the individuals for whom the data were observed. The posterior distributions of the unobserved values given the observed data can then be calculated. Since such calculations may be difficult, Little and Rubin (1987) propose a resampling-based approach for their calculation. Like any approach for handling missing data, its validity depends on the correct specification of a model for the process that generated the missing data.

Capture-Recapture Methods

When the population of interest has not been enumerated and a survey of the prevalence of a condition or size of a subgroup in a community is impractical or otherwise unfeasible, the capture-recapture method may be used. This technique derived from the field of population ecology (Stephen, 1996). For example, one can capture mosquitoes, dust them with harmless florescent material, and release them. The proportion of recaptured mosquitoes in a day or two (allowing sufficient time for remixing but not allowing time for significant mortality) can be used to estimate the total number of mosquitoes in the local population, assuming random mixing and equal probability of selecting labeled and unlabeled mosquitoes. Similarly, small mammals may be trapped, tagged, and released, and then a second trapping recaptures new and old (tagged) mammals, enabling a population estimate.

In human biology and epidemiology, the completeness of population ascertainment can be indirectly estimated using capture-recapture, as with estimations of persons who need HIV therapy, drug addiction services, or other social or medical services. Thus, persons must be “captured” and “marked,” to borrow the ecology model, such that they are available for recapture after release. Sometimes in epidemiology, this is literal, as with prisoners who are injection drug users (IDUs), who are arrested but released after a short time in jail. The proportion who return to jail may be used to estimate the proportion of drug users at risk of being arrested (presumably a large proportion of IDUs); combined with population HIV estimates, the absolute number of HIV-infected drug users can be estimated (Drucker and Vermund, 1989; Dunn and Ferri, 1999).

One may estimate the size of a population from just two samples or through multiple samples. Capture histories may be analyzed to estimate migration, life span, or size in the population of interest. A simple formula reflects the core principle of the basic capture-recapture approach. This simple model requires strong assumptions such as full mixing of persons who have been “captured” and “released” (as with hospitalized patients who go home) with the general population. The time-to-recapture estimation must be long enough to permit remixing and short enough for estimation to be relatively unaffected by deaths, out- and in-migrations, and failure to identify “marks.” The latter may occur, for example, when rehospitalized patients use different names when entering an institution. If the assumptions are met, the formula is expressed as: N = MC/R (where N = total population size estimated; M = total number of persons “captured” and “marked” [i.e., identified] on the first occasion; C = total number of persons “captured” on the second visit; and R = number of identified persons “marked” from the first occasion that were then reidentified on the second occasion) (Chao et al., 2001; Hook and Regal, 1995; International Working Group for Disease Monitoring and Forecasting, 1995a,b; Stephen, 1996).

Marginal Structural Methods and Models

Marginal structural models estimate treatment or intervention effects in observational studies by statistical strategies of controlling for selection bias and confounding variables (Robins, 1999). The fundamental concept behind marginal structural modeling can be explained as follows: Suppose one wishes to compare exposures A and B, which may vary over time, in some population and suppose that, at each time, one could create an identical copy of each study subject. If the actual subject had exposure A at a given time, we would give the copy exposure B and vice versa. One could then compare each subject to his or her copy. We refer to the outcomes for each of the imaginary copies as “counterfactuals” and treat them as missing data. Inverse-probability weighting (IPW) is an approach to handling missing data that reweights observations by the inverse of the probability that they are made. Marginal structural models use IPW to deal with the unobserved (“missing”) counterfactuals. Using IPW and marginal structural model procedures reweight data sets so that treatment and covariates are not confounded.

In “confounding by indication,” an “exposure” is linked to a true causal exposure (e.g., condom use and commercial sex work) but does not itself contribute to the outcome. For example, condom use may be statistically and positively linked to HIV risk, which is counterintuitive (Holmes et al., 2004), but this association may arise because of confounding by indication (e.g., disproportionate use of condoms by sex workers in the population studied). When this occurs in an observational study, the association of the putative risk factor cannot be accurately attributed to the outcome of interest unless one has measured all of the exposures and the relevant confounding factors.

Estimation of causality must take into account time-dependent confounding, and marginal structural models can address selection bias and/or confounding in such analyses. However, inclusion of such factors as time-varying covariates in longitudinal models does not correct for this bias. Such bias occurs most often when “(1) conditional on past treatment history, a time-dependent variable is a predictor of the subsequent outcome and [is] also a predictor of subsequent treatment; and (2) [when] past treatment history is an independent predictor of the time-dependent variable” (Suarez et al., 2008).

Marginal structural models can be used for causal inference even from nonexperimental designs, comparing treatments or interventions, as long as information is reasonably accurate, all confounders are measured, and censoring either is noninformative or can be modeled accurately as a function of known covariates. Better control of confounding than available from simple parametric regression models alone may bring some observational data closer to values that would be measured in a randomized controlled clinical trial. One recent example comes from a study showing that hormonal contraception is a risk factor for HIV acquisition in African women (Heffron et al., 2011). Marginal structural model analyses were used to assess the validity of the Cox proportional hazards regression from this large observational couples study.

Here the committee describes only a few of the challenges that arise in the use of observational data to make inferences about outcomes or service coverage (Teresi, 1994) and the approaches to dealing with them. Nonetheless, the committee seeks to illustrate a few modern statistical methods to make surveillance and programmatic data more useful for evaluation purposes and to illustrate the inherent challenges, both to data collection and to analyses. Correct application of these and other relevant techniques can improve the chances that inferences drawn from imperfect data are valid.

Analysis of Indicators Involving Small Subgroups of People Living with HIV/AIDS

Tracking reductions in HIV-related health disparities will require analysis of indicators by race and ethnicity, sexual orientation, and other demographic variables. The NHAS is aimed at improving access to care and health outcomes for PLWHA and reducing HIV-related health disparities at the national level. Yet, analysis of indicators may occur at a local level, such as to disseminate information to local health departments and HIV care providers on the status of the HIV epidemic in their jurisdictions. In some communities of the United States, the number of individuals who comprise a specific demographic group (e.g., racial and ethnic minority men who have sex with men) may be small. Because the statistical power of an indicator estimate is linked to the number of observations in a sample, small subgroups limit the precision of estimates of care indicators and the ability to compare them with other subpopulations of PLWHA. In epidemiologic studies, investigators may have little choice but to pool very small subpopulations with the larger study population because there is insufficient power to extract the effects of defining subgroup characteristics. With respect to the NHAS, however, this would defeat the purpose of using indicators to track improvements in HIV-related disparities.

Statistical methods for inference may be used for the analysis of indicators involving small subgroups of PLWHA. In general, Bayesian methods are useful for combining information about prevalence, incidence, or treatment effects across different population subgroups (Han and Chaloner, 2005). Group-specific Bayesian estimates are “shrunk” toward (moved closer to) the mean of the quantity of interest over the population included in the combined data set. Because the amount of shrinkage depends on the amount of available information, the smaller the size of the subgroup, the greater will be the reliance on the estimate of the mean. In addition, group-specific Bayesian estimates are sensitive to assumptions regarding the distributions of the random effects; the most common approach of assuming normal distributions leads to the greatest shrinkage. Using other types of random effects distributions, such as Student t or mixtures, can reduce the amount of shrinkage, since they have longer tails and, therefore, allow for a greater probability of outlying values. As an alternative, one can base inference on nonparametric approaches, which can achieve the same goal. Posterior distributions may also tend to be flatter—implying lower precision in estimates—because the strong normal assumption can convey a sense that there is more information on which to base inference than is truly the case if the distributions are nonnormal. While Bayesian methods provide posterior distributions for any subgroup, no matter its size, the inference for that group will rest most heavily on the mean and on underlying assumptions if the subgroup is small. More advanced statistical methods, such as those that do not require parametric assumptions for the distributions of the random effects, can provide more reliable and robust results in this setting.

Growing numbers of studies indicate that social status modifiers such as race and ethnicity, nativity (place of birth), sexual orientation, geographic location, and drug use status often have an impact on important measures of HIV care (e.g., Kempf et al., 2010; Lillie-Blanton et al., 2009; McGowan et al., 2011). For these subpopulations, among whom social status contributes to their risk environment (Farley, 2006; Rhodes, 2009) and treatment access, assumptions of normality of the distribution of random effects may be especially problematic, and approaches that allow for the existence of outliers are particularly needed. One potential consequence of overshrinkage is underestimation of the impact of indicators of social status, such as geographic location, economic status, or drug status, on care experiences. And so although parametric approaches can provide some care data on individuals in these subpopulation groups, they lie within a more restrictive set of assumptions that could temper the use of the results for policy changes.

Epidemiologic studies are an important source of data on care and supportive services received by PLWHA. Health research in general has historically been plagued by an inability to recruit and retain large numbers of racial and ethnic and socioeconomically diverse populations, particularly of sexual minorities or the homeless (Levkoff and Sanchez, 2003; Moreno-John et al., 2004; Sengupta et al., 2000), although studies of PLWHA may do better than studies of other populations in terms of representativeness. Although helpful, statistical methods cannot make up for a lack of sufficient data to estimate indicators for small populations of PLWHA. The development of precise indicator estimates would be facilitated to a greater degree by inclusion of those groups in greater numbers in HIV-related studies.

In a given community, there may be subpopulations of PLWHA that are small in number and have complex health care and supportive service needs for whom the ability to maintain health care regimens depends on access to supportive services. Improvements in linkages between data systems that collect information on clinical care and those that collect information on supportive services (e.g., housing and transportation services) would help to ensure the availability of the full range of data needed to estimate indicators for these subpopulations. Although data system linkages will not address the problem of low statistical power in analyses designed specifically to provide estimates for small subpopulations of PLWHA, nonparametric methods can be used to provide some insights into care needs.

Increased support for training of HIV/AIDS researchers in statistics and methodologies may facilitate the development of expertise in the analysis of data for subpopulations of PLWHA. Such investment could speed the provision of effective treatment to all communities and thereby improve control of HIV transmission.

Audience

Defining and understanding the target audience is one of the first steps in developing a plan for data dissemination (Marriott et al., 2000; Sofaer and Hibbard, 2010b). Potential audiences for data derived from the full set of HIV care indicators identified by the committee already have been identified (federal and state agencies and policy makers, state and local health departments, health care systems, individual providers, consumers [patients], and academic researchers). Selection of the appropriate audience involves consideration of what the data show, the purpose for which the data are being disseminated, and the message that is to be conveyed.

Federal, state, or local policy makers and agencies would be the primary target audience/s if the purpose is to increase or redirect the allocation of resources or to affect policy changes, including the development of new programs to address specific areas of need. Such programs could focus on points in the HIV care continuum that the data might indicate are particularly problematic (e.g., continuity of care) or mediators known to affect those areas (e.g., access to stable housing). Information about improvements on indicators would be useful as well, by showing which current policies and programs are working.

Public and private health care systems, as well as individual providers, might be interested in the data for the purpose of evaluation of, and possible changes in, the HIV care they provide. Such information could permit systems and providers to identify their areas of strength, as well as areas for improvement, in the provision of quality HIV care. Research indicates that dissemination of clinical practice guidelines alone has a minimal effect on provider knowledge and performance, while combination strategies, including those with an education component, are more effective (Marriott et al., 2000). Results of performance indicators also may be more effective in changing provider behavior (Marriott et al., 2000).

PLWHA, and advocacy groups for PLWHA, are other potential audiences for the information on indicators. The information could be used to educate individuals regarding areas in which increased attention and advocacy could improve HIV care.

Depending on what the data show, the dissemination process might target any of these general audiences or a more specific audience within a group, such as policy makers representing a particular region of the United States, HIV care providers who serve patients in a specific demographic group, or patients of a particular race or ethnicity. Ultimately, audience selection should depend on the applicability of the data for that audience and the purpose the data serve (Sofaer and Hibbard, 2010b). Once the audience is defined, the message and the remainder of the dissemination process should be geared to that audience (CDC, 2009).

Definition and Presentation of the Message

Another critical consideration in effective data dissemination is the message to be conveyed. Many audiences are not equipped to understand and process vast quantities of data (Sofaer and Hibbard, 2010a). Data provided without the expertise to interpret them can cause more harm than good. Even the language used to present the information may result in unanticipated misinterpretation (Hibbard and Sofaer, 2010). It is important therefore for an agency to have a clear understanding of the message it wants to transmit and then relay that message to the target audience clearly and concisely, along with the data to support it (Hibbard and Sofaer, 2010; Marriott et al., 2000; Sofaer and Hibbard, 2010a). The details of the message may vary depending on the target audience, as will the way in which the message is presented.

Presentation of the message in the most appropriate way for the target audience is critical to ensure that the message the agency wants to convey is the one that is received by the audience (CDC, 2009; Marriott et al., 2000). Considerations of health literacy and numeracy are important when preparing information for dissemination (Hibbard and Sofaer, 2010; IOM, 2004). Presentations of data from the HIV care and supportive services indicators and trends in the quality of HIV care over time will use different language depending on the audience (e.g., clinical care professionals, policy makers, program administrators, members of the public). Clinical indicators of HIV care that are fully comprehensible to HIV care providers may be incomprehensible to patients or to policy makers. It is important to make the information relevant to what the audience understands and the purpose for which it will use the data. Three papers on “best practices in public reporting” on health care performance data, prepared for the Agency for Healthcare Research and Quality, discuss a number of the pitfalls in and solutions to presenting performance data to health care consumers (Hibbard and Sofaer, 2010; Sofaer and Hibbard, 2010a,b). Although the papers focus on a specific type of data and target audience, the concepts presented may be generalized to other audiences and types of information.

Data Quality and Interpretation

As discussed, the IQA mandates that federal agencies develop quality assurance guidelines for information releases to the public, and a number of HHS agencies have issued Guidelines for Ensuring the Quality of Information Disseminated to the Public. Although it is important for agencies to present the message clearly, concisely, and in language that is understood by and resonates with the target audience, it is also important that they include information about the quality of the data that support the message and the methods used to interpret them (Sofaer and Hibbard, 2010a).

Factors affecting data quality include the source of the data, quality within the system, coverage of the data, confidence range, use of proxies, and analytic methodology applied. The challenge lies in providing sufficient information to permit independent assessment of the data, while not overwhelming the target audience with information that it cannot or will not use (Marriott et al., 2000; Sofaer and Hibbard, 2010a). One approach is to include with the disseminated information a summary, presented in language accessible to the target audience, of the data and the data analysis, including discussion of limitations or gaps in the data and any other relevant information that would enhance the audience’s understanding and evaluation of the data (HHS, 2006b; Marriott et al., 2000; Sofaer and Hibbard, 2010a). At the same time, the agency could make available to interested parties full information on the data set and the methodologies used to assess it (Sofaer and Hibbard, 2010a). CDC, for example, clearly documents and makes publicly available the statistical processes and methodologies used to derive published information, which allows independent statisticians to replicate the results (HHS, 2006b).

Evidence suggests that an audience’s acceptance of data is affected by its perception of the credibility of the data source and the source reporting the information (e.g., professional medical journal versus popular press), as well as proximity of the source to the target audience (Marriott et al., 2000; Sofaer and Hibbard, 2010a). Information intermediaries can help in this regard. Engagement with organizations knowledgeable about and trusted by the target audience may assist in the dissemination of information and help to support the credibility of the information and its source (Sofaer and Hibbard, 2010b).

Methods of Dissemination

A final consideration for effective and efficient data dissemination is selection of the most appropriate and cost-effective method of dissemination. As previously mentioned, federal agencies have a variety of dissemination methods at their disposal, including traditional print media (e.g., reports, peer-reviewed articles, fact sheets, newsletters), electronic media (e.g., websites, podcasts), and public forums (e.g., conferences, planned meetings) (CDC, 2009), and frequently more than one method may be employed.

The target audience and the message and data to be conveyed are factors in the choice of dissemination method. An agency might choose to prepare a report or paper for a peer-reviewed professional journal if the goal is transmit the information to health care systems or providers. Reports, newsletters, and fact sheets might be more effective in reaching policy makers or other agencies. Websites will reach a larger and broader audience, including members of the public. The type of information and style of presentation used for a professional journal will differ markedly from that prepared for dissemination on the agency website. The speed or urgency with which a message needs to be conveyed is another consideration in the selection of dissemination method.