TRIAL OBJECTIVES

Questions such as whether to continue to collect trial outcome data after a participant has discontinued use of the study treatment, whether to use a single or composite outcome measure, how long to measure outcome data, all depend on the estimation goal of the trial. This estimation includes not only the outcome of interest, but also whether the focus is on short- or long-term effects of the intervention and the target population of interest. Possibilities for the latter include the “intent-to-treat” population, or the population of treatment compliers. Before selecting a trial design, it is important to decide on the primary parameter and population of interest, the “causal estimand.” Once the estimand is decided, the clinical trial design can be optimized for the measurement of that estimand.

Recommendation 1: The trial protocol should explicitly define (a) the objective(s) of the trial; (b) the associated primary outcome or outcomes; (c) how, when, and on whom the outcome or outcomes will be measured; and (d) the measures of intervention effects, that is, the causal estimands of primary interest. These measures should be meaningful for all study participants, and estimable with minimal assumptions. Concerning the latter, the protocol should address the potential impact and treatment of missing data.

REDUCING DROPOUTS THROUGH TRIAL DESIGN

The interpretation of the trial findings is more difficult when participants discontinue their assigned interventions before the end of the study. Therefore, the trial design should be selected to maximize the number of participants who are maintained on the study intervention throughout the duration of the trial.

Recommendation 2: Investigators, sponsors, and regulators should design clinical trials consistent with the goal of maximizing the number of participants who are maintained on the protocol-specified intervention until the outcome data are collected.

There is a key distinction between treatment dropout and analysis dropout, and although there are trials in which treatment dropout will understandably be substantial, there is very little reason for substantial amount of missing data, that is, analysis dropouts. Furthermore, for many trial estimands, the benefits of retaining participants in the study can be substantial, including to support an analysis of effectiveness (comparison of treatment policies) and to be able to monitor side effects that occur after discontinuation of treatment.

Recommendation 3: Trial sponsors should continue to collect information on key outcomes on participants who discontinue their protocol-specified intervention in the course of the study, except in those cases for which a compelling cost-benefit analysis argues otherwise, and this information should be recorded and used in the analysis.

Recommendation 4: The trial design team should consider whether participants who discontinue the protocol intervention should have access to and be encouraged to use specific alternative treatments. Such treatments should be specified in the study protocol.

Recommendation 5: Data collection and information about all relevant treatments and key covariates should be recorded for all initial study participants, whether or not participants received the intervention specified in the protocol.

REDUCING DROPOUTS THROUGH TRIAL CONDUCT

In addition to trial design, aspects of trial conduct can also substantially reduce the amount of missing data. Chapter 3 outlines specific trial conduct techniques that should be considered. Given the importance of reducing the frequency of missing data, the monitoring of missing data from the design stage throughout the conduct of a trial needs to be accounted for in the trial protocol.

Recommendation 6: Study sponsors should explicitly anticipate potential problems of missing data. In particular, the trial protocol should contain a section that addresses missing data issues, including the anticipated amount of missing data, and steps taken in trial design and trial conduct to monitor and limit the impact of missing data.

Recommendation 7: Informed consent documents should emphasize the importance of collecting outcome data from individuals who choose to discontinue treatment during the study, and they should encourage participants to provide this information whether or not they complete the anticipated course of study treatment.

Recommendation 8: All trial protocols should recognize the importance of minimizing the amount of missing data, and, in particular, they should set a minimum rate of completeness for the primary outcome(s), based on what has been achievable in similar past trials.

TREATING MISSING DATA

Missing data are often unavoidable, despite best efforts to reduce their occurrence in trial design and conduct. The validity of assumptions concerning the source of missing data can only be assessed jointly by both data analysts and clinicians. Therefore, it is important that the assumptions underlying any selected analysis technique be clearly articulated so that they can be evaluated by clinicians as well as by statistical analysts.

Recommendation 9: Statistical methods for handling missing data should be specified by clinical trial sponsors in study protocols, and their associated assumptions stated in a way that can be understood by clinicians.

Methods like last observation carried forward (LOCF) and baseline observation carried forward (BOCF) are commonly applied in situations in which their underlying assumptions are unrealistic. The analysis methods used should yield confidence intervals for the treatment effect that have the claimed coverage properties and tests should have their nominal size when data are missing.

Recommendation 10: Single imputation methods like last observation carried forward and baseline observation carried forward should not be used as the primary approach to the treatment of missing data unless the assumptions that underlie them are scientifically justified.

Recommendation 11: Parametric models in general, and random effects models in particular, should be used with caution, with all their assumptions clearly spelled out and justified. Models relying on parametric assumptions should be accompanied by goodness-of-fit procedures.

Recommendation 12: It is important that the primary analysis of the data from a clinical trial should account for the uncertainty attributable to missing data, so that under the stated missing data assumptions the associated significance tests have valid type I error rates and the confidence intervals have the nominal coverage properties. For inverse probability weighting and maximum likelihood methods, this analysis can be accomplished by appropriate computation of standard errors, using either asymptotic results or the bootstrap. For imputation, it is necessary to use appropriate rules for multiply imputing missing responses and combining results across imputed datasets because single imputation does not account for all sources of variability.

Recommendation 13: Weighted generalized estimating equations methods should be more widely used in settings when missing at random can be well justified and a stable weight model can be determined, as a possibly useful alternative to parametric modeling.

One very useful source of information that appears to have been rarely used is the follow-up of a sample of participants who withdrew from a study. Such data could be very useful in determining reasons for withdrawal and their missing outcome measurements.

Recommendation 14: When substantial missing data are anticipated, auxiliary information should be collected that is believed to be associated with reasons for missing values and with the outcomes of interest. This could improve the primary analysis through use of a more appropriate missing at random model or help to carry out sensitivity analyses to assess the impact of missing data on estimates of treatment differences. In addition, investigators should seriously consider following up all or a random sample of trial dropouts, who have not withdrawn consent, to ask them to indicate why they dropped out of the study, and, if they are willing, to collect outcome measurements from them.

Given that the assumptions for the missing data mechanism cannot be validated, the sensitivity of inferences for treatment effects in clinical trials to those assumptions needs to be assessed.

Recommendation 15: Sensitivity analyses should be part of the primary reporting of findings from clinical trials. Examining sensitivity to the assumptions about the missing data mechanism should be a mandatory component of reporting.

UNDERSTANDING THE CAUSES AND DEGREE OF DROPOUTS IN CLINICAL TRIALS

A crucial issue that sponsors must wrestle with in planning a clinical trial is how much missing data they are likely to experience, how much could be reduced through the use of various techniques (such as those outlined in this report), and consequently if they implement these various techniques, what degree of missingness is likely to remain. The answers to these questions will help trial sponsors decide on how to plan for missing data when determining sample size, whether steps are needed to reduce the amount of missing data (some of which may be resource intensive), and the potential for lack of robustness of final estimates of intervention effects to assumptions about missing data. In addition, analysts need to know what assumptions about the missing data mechanisms are scientifically defensible.

Information from previously collected clinical studies would help in answering these questions. Although FDA retains data from all clinical trials over which it has oversight, the data are confidential to the company that sponsored the trial and are therefore not shared. And although there is some research on why participants drop out of different kinds of clinical trials, empirical evidence is lacking for many types of trials. There is a need for more standardized data collection, documentation, and analysis of the reasons for and the frequency of missing data. Systematic investigations of factors related to treatment dropout and withdrawal and to missing data more generally are needed.

A pharmaceutical company that has been researching interventions in a particular area for a long time may have internal data that can provide some of this information. However, if a company is small or has limited prior experience, having access to information from prior clinical trials would be extremely useful in trial design.

Recommendation 16: The U.S. Food and Drug Administration and the National Institutes of Health should make use of their extensive clinical trial databases to carry out a program of research, both internal and external, to identify common rates and causes of missing data in different domains and how different models perform in different settings. The results of such research can be used to inform future study designs and protocols.

While it is difficult to be specific, characteristics of a trial that has failed because of missing data concerns include (a) rates of missing data that are two to three times as large as the difference in rate of successful outcome between the groups; (b) differential rates of missing data across treatment arms, indicating a high likelihood that biases would not cancel out in the treatment comparison; (c) lack of a record about the reasons for missing data, making it unclear whether the mechanisms are MAR; (d) lack of auxiliary data that would be the basis for plausible missing data adjustments; (e) marginal treatment effects that might be easily overturned by uncertainty from missing data; and (f) inadequate analysis methods that do not reflect uncertainty from missing data.

Many of the analysis techniques described in Chapters 4 and 5 have been explored both theoretically and in applications in the research literature over the past 20 years. However, their applications to clinical trials have been limited. There seems to be a reticence on the part of biostatisticians, both at the drug and device companies and at FDA, to embrace these various techniques. We conjecture that this reticence may be a by-product of the regulatory environment, a result of the limited development of supporting software for newer methods, or a result of a lack of training and education. We believe that once these techniques are implemented in a few major clinical trials, any conservatism from the regulatory environment would be offset by the more effective use of the information provided by these techniques. Also, as pointed out in Chapters 4 and 5, software now exists for most of the techniques as part of the most commonly used statistical packages. The remaining possibility, training and education, can be addressed by making education at FDA a higher priority. It is also important that FDA’s clinical reviewers have some understanding of modern analysis methods so that they can assist in the judgment as to the reasonableness of assumed missing data mechanisms.

Recommendation 17: The U.S. Food and Drug Administration (FDA) and drug, device, and biologic companies that sponsor clinical trials should carry out continued training of their analysts to keep abreast of up-to-date techniques for missing data analysis. FDA should also encourage continued training of their clinical reviewers to make them broadly familiar with missing data terminology and missing data methods.

Throughout this report, we have advocated that further research be carried out in a number of important areas. We have decided to bring together those calls for additional research here in this final chapter. Areas in need of further research include

(1)

designs for the follow-up of participants in clinical trials who have dropped out of the study (e.g., referred to here as analysis dropouts) and who have not withdrawn their consent,

(2)

collecting the typical rates and likely causes of missing data in various kinds of clinical trials,

(3)

the effect of missing data on the power of clinical trials,

(4)

how to set useful target rates and acceptable rates of missing data in clinical trials,

(5)

the robustness of missing data methods such as inverse probability weighting methods and multiple imputation methods to assumptions,

(6)

the assessment of goodness-of-fit for the parametric models used to analyze data from clinical trials (when there is missing data),

(7)

the performance of double-robust procedures in comparison to more commonly used procedures,

(8)

the impact of missingness in auxiliary variables on the various current methods, and ways of reducing the associated bias,

(9)

methods of sensitivity analysis in clinical trials, particularly for nonmonotone patterns in longitudinal data,

(10)

methods for assessing and limiting the impact of informative censoring for time-to-event outcomes, and

(11)

how to develop effective decision rules based on the input from sensitivity analyses.

We have collected the highest priority of these calls for additional research in a final recommendation, adding to that a call for the development of the associated software tools.

Recommendation 18: The treatment of missing data in clinical trials, being a crucial issue, should have a higher priority for sponsors of statistical research, such as the National Institutes of Health and the National Science Foundation. There remain several important areas where progress is particularly needed, namely: (1) methods for sensitivity analysis and principled decision making based on the results from sensitivity analyses, (2) analysis of data where the missingness pattern is non-monotone, (3) sample size calculations in the presence of missing data, and (4) design of clinical trials, in particular plans for follow-up after treatment discontinuation (degree of sampling, how many attempts are made, etc.), and (5) doable robust methods, to more clearly understand their strengths and vulnerabilities in practical settings. The development of software that supports coherent missing data analyses is also a high priority.