Hormone Replacement Therapy

The first case was presented by JoAnn E. Manson from Harvard Medical School on the impact of hormone replacement therapy (HRT) on health. Both observational studies and clinical trials have contributed critically important information to elucidate the health effects of HRT with estrogen and progestin and to inform decision making, and they constitute a model suggesting that research findings should be considered in the context of the totality of available evidence and that studies should be designed to complement and extend existing data. Manson noted that observational studies and randomized clinical trials of menopausal HRT and coronary heart disease (CHD) have produced widely divergent results. Observational studies had suggested a 40–50 percent reduction in the risk of CHD among women taking HRT, whereas randomized trials suggested a neutral or even elevated risk of coronary events. Well-recognized limitations of observational studies, including the potential for confounding by lifestyle practices, socioeconomic status, education, and access to medical care, as well as selection factors related to “indications for use,” can explain only some of the discrepancies. Other methodologic factors that may help to explain the differences include the limitations of observational studies in assessing short-term or acute risks, which led to incomplete capture of early clinical events after therapy began, and the predominance of follow-up time among compliant long-term users of HRT. In contrast to the greater weighting of long-term use in observational studies, clinical trial results tend to reflect shorter term use. Given that CHD risks related to HRT are highest soon after initiation of therapy, these differences may contribute substantially to the discrepancies observed.

Methodologic differences between observational studies and clinical trials, however, may not fully elucidate the basis for the discrepancies observed. The findings of observational studies and clinical trials are remarkably concordant for other health outcomes, including stroke, venous thromboembolism (blood clot), breast cancer, colorectal cancer, and fracture—outcomes that also should be affected by confounding and selection biases. Indeed, an emerging body of research suggests that the age of menopause or time since menopause critically influences the relationship between HRT and CHD outcomes.

Importantly, observational studies should be designed to capture both short- and long-term risks and should have frequent updating of exposure variables of interest (electronic health and pharmacy records may be useful). Clinical trials must be powered adequately to assess clinically relevant subgroups and to address the possibility of a modulating effect of key clinical variables. Consideration of absolute risks in research presentation and interpretation is critically important. Finally, it may be helpful to incorporate intermediate and surrogate markers into study designs, although such markers can never fully replace clinical event ascertainment.

Drug-Eluting Coronary Stents

Research to date strongly suggests that further understanding of and solutions to the safety issues concerning drug-eluting stents (DES) will likely come from a mix of randomized trials and observational registries conducted in both the premarket and postmarket arenas and a collaborative effort among regulators, industry, and academia. As Ashley B. Boam from the FDA recounted, coronary drug-eluting stents have dramatically changed interventional cardiology practice since their introduction in the United States in 2003. These products—a combination of a metal stent and an antiproliferative drug—have significantly reduced the need for reintervention compared to the previous standard of care, bare metal stents. This substantial improvement has led to widespread adoption of these products and use in patients outside those enrolled in the initial pivotal clinical studies. The desire to bring additional DES technology to the market quickly drove research into the identification of potential surrogate markers for effectiveness. While at least two measures obtained from angiography (i.e., late loss and percentage diameter stenosis of vessel) have been identified as biomarkers with a strong correlation to clinical effectiveness—specifically, the need for a reintervention—no such marker has been identified as a possible surrogate for safety outcomes, such as death or myocardial infarction.

In the last half of 2006 and through 2007, however, the emergence of stent thrombosis—the occurrence of a clot within the stent that often leads to myocardial infarction or death—shifted the focus of DES research significantly from effectiveness to safety. Recent meta-analyses and research from centers in Europe and the United States have indicated that late stent thrombosis may be an ongoing risk to DES patients. Low event rates, less than 1 percent, and late-term occurrence (beyond 1 year postimplantation) have complicated efforts to understand the true incidence and etiology of this noteworthy complication. Confounding the picture is the lack of appropriate studies to optimize prescription of mandatory adjunctive dual antiplatelet therapy, early interruption of which is one known risk factor for stent thrombosis. The issue is challenging in part because what is under consideration is a low frequency event with a late-term appearance, which generally mandates very large and long studies. Randomization, long-term follow-up, concurrent device iterations and new platforms, on-label versus broad clinical use considerations, and the role of resistance to acetylsalicylic acid and/or clopidogrel are among the other challenges.

Bariatric Surgery

Population-based registries—appropriately funded and constructed with clinician engagement—offer a compromise of strengths and limitations and may be the most effective tool for evaluating emerging healthcare technology, argued David R. Flum of the University of Washington. The dichotomy between “effectiveness” and “efficacy” is particularly relevant in evaluating surgical interventions where characteristics of the surgeon (experience, training, and specialty), variations in technical performance, patient selection, practice environments, and publication bias all influence the understanding of healthcare interventions. Case series, often authored by experts in the field, dominate the surgical literature. Despite their limitations, they are strong influences on clinical and policy decisions.

Bariatric interventions include a group of operations that have become increasingly popular with the advent of less-invasive surgical approaches and epidemic obesity. Understanding the safety and efficacy of bariatric interventions has come almost exclusively through single-center case series. A research group based at the University of Washington has worked to expand knowledge in this field through the use of retrospective population-level cohorts using administrative data, clinical registries, and longitudinal prospective cohorts that work to assess effectiveness. These safety data have been helpful in coverage decisions by payers in assessing quality improvement opportunities and in providing more realistic assessments of these interventions. Inherent limitations in effectiveness research include tradeoffs among numbers of patients and details on patients, the granularity and accuracy of the data, and limits to the types of outcomes that can be evaluated (i.e., no quality-of-life or functional data).

Antipsychotic Therapeutics

As shown in recent research to ensure that antipsychotic medication use is clinically effective, advances in study designs, databases, and analytic methods provide a toolbox of complementary techniques from which researchers can draw, suggested Phillip S. Wang of the National Institute of Mental Health (NIMH). Antipsychotic medications are now widely used by patients and account for a large proportion of pharmaceutical spending, particularly in public healthcare programs. However, there is a paucity of evidence to help guide clinical, purchasing, and policy decisions regarding antipsychotic medications. The recently completed, NIMH-sponsored comparative effectiveness trials of antipsychotic medications in patients with schizophrenia (the CATIE trial), for example, blends features of efficacy studies and large, simple trials to create a pragmatic trial to provide extensive information about antipsychotic drug effectiveness over a drug course of at least 18 months. Recent advances in databases, designs, and methods can also be brought to bear to improve antipsychotic effectiveness. New study populations and databases have been developed, including practice-based networks that look at psychiatric care. Large administrative datasets are also available, such as Medicaid and useful health maintenance organization (HMO) databases, which are ideal for studying primary care in the setting where most mental health care is actually received.

Other approaches can be employed when trial data are not available. Researchers can use clinical epidemiologic data and methods, which are often a useful addition to the literature. When trials and quasi-experimental and even epidemiologic studies are not possible, researchers can use simulation methods. In addition, researchers have developed new means to deal with threats to validity—both threats to external validity, as in the development of effectiveness research, and threats to internal validity. These include new analytic methods, propensity score adjustments, and instrumental variable techniques.

Cancer Screening

Developing comparative effectiveness information about screening tests is a complex undertaking, as demonstrated by Peter B. Bach of Memorial Sloan-Kettering Cancer Center. The general rationale for screening in the context of clinical medicine or typical practice is that clinical disease usually “presents”—patients arrive with symptoms or signs that define a population that can be screened. Such circumstances actually make a strong argument for looking for preclinical conditions. That is what screening is intended to do—essentially scan an unaffected population to look for people who are at risk for developing some condition. The underlying principle of such an investigation is that theoretically we can decrease morbidity and mortality and other negative outcomes by looking for patients with preclinical conditions. Screening is widely encouraged, and one could argue that it is the dominant activity in much of primary care. Most medical journals, for example, regularly publish tables of screening evaluations, which are lists of questions that physicians should ask their patients to determine their risk for given diseases.

How these screenings impact a patient’s health need to be understood better. A recent study found that screening for lung cancer with low-dose computed tomography may increase the rate of lung cancer diagnosis and treatment, but may not meaningfully reduce the risk of advanced lung cancer or death. Until more conclusive data are available, asymptomatic individuals should not be screened outside of clinical research studies that have a reasonable likelihood of further clarifying the potential benefits and risks. There are similar examples—in prostate cancer, in breast cancer with mammography, in renal cell cancer, and in melanoma. A paradoxical reality of the surrogate end-points often used to evaluate effectiveness in these cases is that they are readily available, but can be misleading. Simply stated, screening can often pick up pseudodisease, conditions that are not significant, yet trigger interventions or conditions that cannot be cured that are too often quickly characterized as “early” or “curable.” Refuting the principle of “catch it early” is difficult, but there are many approaches to this goal, each with its own advantages and disadvantages.

Innovative Approaches to Trials

Clinical trials play an important role in assessing the effects of medical interventions, in particular where observational studies are often inadequate, such as the detection of modest treatment effects or when the risk of an invalid answer is substantial. Robert Califf from Duke University emphasizes the importance of focusing discussion about medical evidence on a serious examination of ways to improve the operational methods of both approaches and of building human systems that take advantage of the power of modern informatics on improving both RCTs and observational studies. In particular, the design and conduct of RCTs needs to evolve to take further advantage of modern informatics and to provide a more flexible and practical tool for clinical effectiveness research. Improvements in the structure, strategy, conduct, analysis, and reporting of RCTs can help to address their perceived limitations related to cost, timeliness, and reduced generalizability.

Innovative approaches discussed including the conduct of trials within “constant registries” and targeting the standard set of rules for the conduct of trials to make them more adaptable and customized to meet research needs. A relevant model is the clinical practice database found at The Society of Thoracic Surgeons. This database has been used for quality reporting and increasingly to evaluate operative issues and technique. An extension of this model would be to develop constant disease registries capable of drawing on multiple EHR systems. The conduct of RCTs within the database would allow researchers to revolutionize the timeframe and costs of clinical trials. Trials also could take better advantage of “natural units of care” with cluster randomization, or provide information more relevant to practice by focusing on research questions based on gaps in clinical practice guidelines or being conducted in real-world practice (e.g., pragmatic clinical trials). Research networks offer the opportunity to enable the needed sharing of protocols, data structures, and other information.

A promising initiative for improvements in the quality and efficiency of clinical trials is the FDA Critical Path public–private partnership: Clinical Trials Transformation Initiative (CTTI). A collaboration of the FDA, industry, academia, patient advocates, and nonacademic clinical researchers, CTTI is designed to enhance regulations that improve the quality of clinical trials, eliminate guidances and practices that increase costs but provide no value, and conduct empirical studies of the value of guidances and practices. Primary barriers to innovation include the lack of appropriately structured financial incentives and the caution toward change that comes with a highly regulated market. To contend with this substantial barrier to ensuring that innovative approaches are implemented in practice, the research community should adopt a model from business of establishing “envelopes of creativity,” or environments in which researchers could innovate with a certain creative freedom, and where they would have appropriate financial incentives.

Califf concludes that smarter trials that provide timely information on outcomes that matter most to patients and clinicians at an acceptable cost will become an integral part of practice in learning health systems as trials increasing become embedded into the information systems that form the basis for clinical practice. These systems also will provide the foundation for integrating modern genomics and molecular medicine into the framework of care.

Innovative Analytic Tools for Large Clinical and Administrative Databases

Because healthcare databases record drug use and some health outcomes for increasingly large populations, they will be a useful data resource for timely, comparative analyses that reflect routine care. Confounding is one of the biggest issues facing effectiveness research in the analyses of these large-claims databases. While recognizing that instrumental variable analyses have the drawback of producing certain levels of untestable assumptions, Sebastian Schneeweiss from Harvard Medical School proposes that their use can lead to substantial research improvements, particularly in situations with strong confounding and where it is likely that important confounders remain unmeasured in a data source. Several developments may bring the field closer to acceptable validity, including approaches that exploit the concepts of proxy variables using high-dimensional propensity scores and exploiting provider variation in prescribing preference using instrumental variable analysis.

Epidemiologists have a number of techniques that can control for confounding by measured factors, but instrumental variables are a promising approach to address unmeasured confounders because they are an unconfounded substitute for the actual treatment. In this approach, instead of modeling treatment and outcome, researchers model the instrument—which is unconfounded on the outcome—and then correct the estimate for the correlation between the instrumental variable and the actual treatment. In this respect valid results require the identification of a quasi-random treatment assignment in the real world, such as interruption in medical practice. Recent work has also demonstrated the potential of provider treatment preference as a random component in the treatment choice process, providing an additional instrument worth consideration for comparative drug effectiveness studies.

Instrumental variable analysis is an underused, but very promising, approach compared to effectiveness research using nonrandomized data, and researchers should routinely explore whether an instrument variable analysis is possible in a particular setting. Additional work is underway to develop better methods to assess its validity and to develop systematic screens for instrument candidates.

Adaptive and Bayesian Approaches to Study Design

Adaptive and, particularly, Bayesian approaches to study design offer opportunities to improve on randomization and to facilitate new ways of learning in health care. Donald A. Berry from the University of Texas M.D. Anderson Cancer Center contends that these approaches can be used to make RCTs more flexible by using data developed during a study to guide its conduct and to incorporate different sources of information to strengthen findings related to comparative effectiveness. The historical, frequentist approach, in which a design must be completely designed in advance and the study must be complete before inferences can be made, impedes the ability to continually assess and alter a course of study based on accrued learning. In contrast, the Bayesian approach’s ability to calculate probabilities of future observations based on previous observations enables an “online learning” ideal for developing adaptive study designs. Prospective building of adaptive study designs is critical and, except in the simplest situations, requires simulation.

Adaptive designs increase the flexibility of RCTs by enabling modifications—based on interim or other study results—including stopping the study early, changing eligibility criteria, expanding or extending accrual, dropping or adding study arms or doses, switching between clinical phases, or shifting focus to subsets of patient populations. These adaptations not only enable rapid learning about relative therapeutic benefits but also improve the overall efficiency of research. Flexibility with respect to patient accrual, for example, may enable a needed increase in study sample size, potentially minimizing the need for additional follow-on studies.

Inherently a synthetic approach, Bayesian analysis can also enhance our capacity to appropriately aggregate information from multifarious sources. For instance, in addition to use in meta-analyses, this approach has been used recently to help answer complex questions, such as the proportional attribution of mammographic screening and adjuvant treatment with tamoxifen and chemotherapy in a drop in breast cancer mortality in the United States. The results of the Bayesian models used to explore this question were consistent with non-Bayesian models as well as those derived in clinical trials.

In conclusion, Berry notes that although the rigor and inflexibility of the current research paradigm has been important to establishing medicine as a science, new approaches such as Bayesian thinking and methodologies can help to move the field even further by making research more nimble and applicable to patient care, while maintaining scientific rigor.

Simulation and Predictive Modeling

Certain research questions or evidence gaps will be difficult or impractical to answer using clinical trial methods. Although physiology-based or mechanistic models have been used only recently in medicine, as noted by Mark S. Roberts from the University of Pittsburgh and representing Archimedes, Inc., physiology-based models, such at the Archimedes model, have the potential to address these gaps by extending results beyond the narrow spectrum of disease or practice found in most trials. These trials can also reduce the cost and time required to complete RCTs.

Physiology-based models aim to replicate disease processes at a biological level—from individual variables to system-level, whole-organ relationships. The behavior of these elements and effect on health outcomes are modeled using equations derived from and calibrated with data from empirical sources. When properly constructed and independently validated, these models not only can serve as useful tools to identify, set priorities in, or facilitate the design of new trials, but also can be engaged to conduct virtual comparative effectiveness trials. When time, cost, or other factors make doing a trial impossible, an independently validated, physiology-based model provides a useful alternative.

Emerging Genetic Information

At the forefront of discovery research, genomewide association studies permit examination of inherited genetic variability at an unprecedented level of resolution. As described by Teri Manolio of the National Human Genome Research Institute, given 500,000 or even a million SNPs (single-nucleotide polymorphism or differences among individuals within species) scattered across the genome, researchers can capture as many as 95 percent of variations in the population. This capacity enables “agnostic” genomewide evaluations, whereby a researcher does not need to preformulate hypotheses or to limit examination to specific candidate genes, but rather can scan the entire genome. Following the availability of high-density genotyping platforms, the pace of genomic discovery has accelerated dramatically for an increasingly broad array of traits and diseases. However, examples of genomewide association studies do have drawbacks. Given the large number of comparisons per study, there is an unprecedented potential for false-positive results. Validation of findings through replication of results generally requires expanding studies from a small initial set of individuals to as many as 50,000 participants.

Two prototypes for applying genomic information from genomewide association studies to clinical effectiveness research are genetic variants related to two traits—Type 2 diabetes risk and warfarin dosing. Though both have sufficient scientific foundations and clinical availability, they remain many steps away from clinical application. Gaining more from these types of insights from genomic research will require additional epidemiologic and genomic information and evidence of impact on outcomes of importance. This evidence can be derived by linking genotypic data to phenotypic characteristics in clinical or research databases, an approach being explored by a number of biorepositories. The National Human Genome Research Institute’s eMERGE network is applying genotyping to subsets of participants in a number of biorepositories with electronic health records. If the phenotypic measures derived from the EHRs are standardized to increase reliability, these types of linked databases hold significant promise for clinical effectiveness research.

Capacity for research, including the testing and interpretation of results, will require significant laboratory infrastructure, including a valid, readily available, FDA-certified, affordable test, conducted under the auspices of a CLIA (Clinical Laboratory Improvement Amendments)-certified laboratory. Robust electronic health records will also be critical to receive data and provide real-time performance feedback so that patients who receive abnormal results can be given suggestions for how to process that information and proceed from that point. In addition, tools for identifying emerging genomic information with potential clinical applications are needed. In this respect, a useful model is the database of Genotype and Phenotype (dbGaP), an accessible but secure large-scale database that receives, archives, and distributes results of studies of genotype–phenotype associations.

Also of vital importance is infrastructure related to policy and educational needs. For example, there is a pressing need to ensure confidentiality and privacy protection, specifically because the potential for discrimination by employers and insurers might occur if they have access to genomic information. To that end, the recently approved Genetic Information Non-Discrimination Act will be helpful. The research community also needs consensus on what should be reported to patients when abnormalities appear—what to tell them, when to tell them, and how to tell them—and adequate consent policies and procedures as well as consistent Institutional Review Board approaches. A flexible approach to genetic counseling is also needed, including the ability for patients to obtain adequate counseling from someone other than a certified genetic counselor. Also needed is a better educational infrastructure to ensure that these issues are discussed both by physicians and by patients, even during the course of ongoing genomic research; this would include better reporting guidelines for both patients and physicians. Education is needed in medical schools and nursing schools, at professional conferences, and in ongoing professional development and training to ensure that caregivers are “genomically literate.” At the same time, we have a responsibility to also educate the general population, so that patients can develop a deeper understanding of genomics. By learning how genomics affects their lives and their health care, patients will know what questions to ask.

The Electronic Health Record and Care Reengineering: Performance Improvement Redefined

Ronald A. Paulus reported on Geisinger’s use of EHRs to transform care delivery to support his contention that there is more potential than currently exploited at the nexus of point-of-care systems and research. When systematically captured, the data produced by these systems can be used to inform and improve clinical practice. Demonstrating the potential of EHRs for impact beyond practice standardization and decision support mechanisms, Geisinger has used these resources in the production of “delivery-based evidence.” EHRs capture data directly relevant to real work practice and can therefore provide extensive, longitudinal data. When coupled with an integrated data warehouse, the creation of a unique data resource can be mined for both clinical and economic insights. These data can also facilitate observational studies that address issues of clinical relevance to complement and fill gaps in RCT data. In developing such models, it is expected that more thought needs to be given to aggregating, transforming, and normalizing data in order to conduct productive analysis that will bridge the knowledge creation gap.

To illustrate the power inherent in linking data to clinical care, Paulus reviewed Geisinger’s work to enhance performance improvement (PI) initiatives. EHRs and associated data aggregation and analysis to complement PI initiatives are increasingly being adopted in the healthcare setting. At Geisinger, PI has evolved into a continuous process involving data generation, performance measurement, and analysis to transform clinical practice. Underlying this transformation is an EHR platform fully used across the system. Geisinger’s integrated database, including EHR, financial, operational, claims, and patient satisfaction data, serves as the foundation of a Clinical Decision Intelligence System (CDIS) and is used to inform and document the results of PI efforts. PI Architecture draws upon CDIS and other inputs (e.g., evidence-based guidelines, third-party benchmarks) and leverages this information via decision support applications to help the organization answer important questions that could not be addressed before.

Key goals supported by PI Architecture include (1) assessment of PI initiatives’ returns on investment (ROIs); (2) simultaneous focus on quality and efficiency; (3) development and refinement of reusable components and modules to support future PI efforts; and (4) elimination of any unnecessary steps in care, automating processes when safe and effective to do so, delegating care to the least cost, competent caregiver, and activating the patient as a participant in her own self-care. The PI Architecture enhances and refines the traditional Plan-Do-Study-Act (PDSA) cycle and yields advantages of reduced cycle time, increased relevance, increased sustainability, increased focus on ROI, and enhanced research capabilities. Key features include (1) use of local data to document the current state of practice and to direct focus on areas of greatest potential improvement; (2) use of electronic record review and simulation to confirm hypotheses and to project the benefits of varying avenues and degrees of change; (3) testing on a small scale, using an iterative approach that builds toward a strategy of rapid escalation; and (4) leveraging of reusable parts from past initiatives to build core infrastructure and accelerate future work.

Administrative Databases in Clinical Effectiveness Research

As described by Alexander M. Walker of Worldwide Health Information Science Consultants and the Harvard School of Public Health, data from health insurance claims form the backbone of many health analytic programs. Although administrative databases are being used more effectively for research, their development and especially their application for generating insights into clinical effectiveness require careful consideration and attention to potential methodologic pitfalls and hazards. Nonetheless, there is extraordinary promise in these resources.

Insurance claims data, derived from government payers or independent health insurers, are comprehensive, population based, and well structured for longitudinal analysis. All services and therapeutics for which there is a payment enter the data system, with easy linkage across providers. The regional or employment-based nature of the populations covered includes medicine as actually provided, not just the care that reflects best practice. The need for multiple providers to interact with multiple insurers or with a government mandate has led to highly standardized data structures supported by regular audit. Finally, these data are available for large numbers of individuals; the largest database with complete information is estimated to include data on 20 million patients.

Although claims data are excellent resources for answering many questions in health services research, they are not always sufficient for clinical research. For example, although labs ordered are recorded, the outcomes or results are not. To address this limitation, research groups have begun to augment their core files—adding laboratory and consumer data, creating the infrastructure for medical record review, implementing methods for automated and quasi-automated examination of masses of data, developing “rapid-cycle” analyses to circumvent the delays of claims processing and adjudication, and opening initiatives to collaborations that respect patients’ and institutions’ legitimate needs for privacy and confidentiality. These enhanced databases provide the information that allows researchers to trace back to specific patients or providers for additional information.

Enhanced claims databases that have been used to support surveillance programs along with automated and quasi-automated database review provide potential decision support tools for clinical safety and efficacy. Basic issues of confounding remain, however, and much attention is needed on these emerging tools to capture the full potential of these databases.

Clinical Effectiveness Research: The Promise of Registries

The dynamic and highly innovative character of healthcare technologies has been important to improvements in health; however, because intervention capacities often evolve due to iterative improvements or expanded use in practice, assessing their effectiveness presents a substantial challenge to researchers and policy makers. Because clinical registries capture information important to understanding the use of diagnostic and therapeutic interventions throughout their lifecycle, they are particularly valuable resources for assessing real-world health and economic outcomes. Alan J. Moskowitz and Annetine Gelijns from Columbia University suggest that in addition to providing information important to guiding decision making for patient care and setting policy, registries are valuable for assessing the performance of physicians and institutions, such as through the use of risk-adjusted volume–outcome relationship studies, and for increasing the efficiency of RCTs.

Several examples of findings derived from registry data on left ventricular assist devices (LVADs) were presented by Moskowitz to illustrate the potential of registries to improve effectiveness research. He pointed out that in contrast to efficacy trials or administrative databases, registries are able to keep pace with the dynamic process of medical innovation. The premarket setting has limits on what can be learned because of the nature of efficacy trials, which are usually short and conducted in narrow populations and under ideal conditions. Once interventions are introduced into general practice, they are used in broader patient populations and under different practice circumstances. For example, only 4 percent of patients treated with coronary artery bypass grafts in practice meet the eligibility criteria of the initial trials (the elderly, females, and those with co-morbidities were excluded). Expanded use of interventions in clinical practice creates a locus for learning and innovation, with the frequent discovery of new and unexpected indications for use, as well as the accrual of knowledge on the appropriate integration of technology into the care of particular patients. Characterized as an “organized system using observational study methods to collect uniform data to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves a predetermined scientific, clinical, or policy purpose,” clinical registries collect data information important to learning and innovation. Examples include long-term outcomes and rare adverse events and outcomes achieved when technology is used by a broadened set of providers or patients. Clinical registries also provide comparative effectiveness information.

Registries offer a powerful means to capture innovation and downstream learning that take place in practice and to develop information complementary and supplementary to that produced by randomized trials. Enhancing the value of registries for clinical research requires improving the quality of data obtained, while decreasing costs and other barriers to data access. Special attention needs to be paid to the definition and standardization of target populations and outcomes (e.g., adverse events); efforts to address bias; measures to ensure representative capture of the population; and sound analytical approaches. Incorporating data collected in the usual course of patient care may help to reduce the burden and cost of registries.

Opportunities to address the traditional weaknesses of registries are presented by advances in informatics, analytical techniques, and new models of financing. The potential of registries to improve the efficiency of randomized trials also must be addressed. The development of investment incentives for stakeholders is important to improving the viability of clinical registries. Although registries have been created by public, not-for-profit, or private organizations, public–private partnerships offer a new model for registry support.

Distributed Data Networks

The variety of information created by healthcare delivery has the potential to provide insights to improve care and support clinical research. Increasingly, these data are held by many organizations in different forms. Although the aggregation of disparate databases into a super dataset may seem desirable for the improved study power and strength of findings provided by larger numbers, such efforts also face significant challenges due to privacy concerns and the proprietary nature of some data. Richard Platt, from Harvard University illustrated how distributed research models circumvent these issues and minimize security risks by allowing the data repositories of multiple parties to remain separately owned and controlled. These models also provide an interface to these stores of highly useful data that allows them to function as a large combined dataset. This approach also takes advantage of local expertise needed to interpret content.

Work at the HMO Research Network has demonstrated the feasibility of this approach for biosurveillance and vaccine safety networks, and several additional models are currently in development. The FDA is calling for the development of a sentinel network to support postmarketing safety research, AHRQ has written a contract to develop a prototype distributed network that will be targeted at comparative effectiveness and safety research, the Robert Wood Johnson Foundation has funded the development of a distributed network that will evaluate quality measures, and the National Institutes of Health (NIH) clinical translational science centers are trying to develop ways for the centers to collaborate with one another in a wide range of clinical research activities. Opportunities to build and use these types of resources also extend to the private sector.

Effective governance models are most pressing for the development and use of these networks. Attention must be given to policies, funding, and further research that supports their development, and to make such work more efficient. Consideration should also be given to developing common standards for distributed research networks so that a single infrastructure might serve multiple functions, even as networks accommodate different kinds of interfaces and governance systems. Overall, governance of such networks, as well as funding for them, will be nontrivial concerns.

Large Data Streams and the Power of Numbers

Data are increasingly generated and captured throughout the research, development, and delivery of healthcare products and services. Sharon-Lise Normand of Harvard Medical School contends that the availability of large data streams holds potential to enhance our capacity to produce clinically useful information, but under the current evidence paradigm, these types of information are often wasted. By treating these data sources as silos of information, opportunities are lost for insights on issues such as treatment heterogeneity, or multiple outcomes and patient subgroups. New analytic strategies are needed to more effectively deploy these resources and improve the efficiency with which the research community produces information.

Taking better advantage of large data streams is important in moving to the next generation of studies. The development and implementation of pooling algorithms are needed, as are inferential tools to detect relationships among diverse data sources and to appropriately combine them. Possible applications include comparative effectiveness research through the enhancement of trial results or inferences derived from single data sources. Moreover, although the research community has some experience in pooling observational data with trial data, study designs that exploit features of the emerging diverse data sources—such as hybrid designs, preference-based designs, and quasi-experimental designs—have not yet been exploited to the full potential.

The number of data sources will only continue to expand, and infrastructure is needed to enable their optimal use, including support for the development of new, innovative analytic strategies and mechanisms for data sharing, documentation, and quality control. Approaches are needed to assess data-pooling strategies, minimize false discoveries, and validate findings. More opportunities are needed to explore approaches to pooling different data sources and employing various study designs. Finally, we need to educate policy makers and researchers in the interpretation of results from the new designs.

Observational Studies

Observational studies add value to the research paradigm through their ability to address the dilemma presented by the costs, slow pace, and other logistical difficulties of conducting RCTs. Additionally, suggested Wayne A. Ray of Vanderbilt University, this approach is essential for answering important clinical questions in which RCTs are not appropriate. Although findings of observational studies are intrinsically more prone to uncertainty than those from randomized trials, abiding by some fundamental epidemiologic principles will allow clinicians to better exploit the wealth of available observational data. Common errors include elementary design mistakes; failure to identify a clinically meaningful t₀, or start to follow-up; exposure and disease misclassification; use of overly broad endpoints for safety studies; confounding by the “healthy drug-user effect”; and marginal sample size. These sources of bias and error can easily lead to the design of a “false-negative” safety study or a “false-positive” efficacy study; upon examination, many controversial or misleading results of observational studies result from such suboptimal methodology.

Although the design of observational studies is a complex subject, opportunities exist to improve their capacity to contribute to clinical effectiveness research. The notion that study design and analysis are quick and inexpensive is misleading. Resources and expertise are needed to support state-of-the-art observational studies—in which consideration is given at the outset of the question at hand and how the various biases might apply. For analyses examining safety, limitations that lead to false results are fairly easy to identify and counteract, and a greater challenge is to address conflicts of interest concerns related to their conduct. For efficacy, a first step is to ensure that the expected benefits of therapy exist for the population as a whole, if not for the individual. Underpinning all of these needed advancements is improved education and training of researchers. Epidemiologists need to become more familiar with the clinical and pharmacological principles that affect the use of observational data and to ensure that these guide the design and conduct of proposed and published studies.

Enhancing the Evidence to Improve Practice: Experimental and Hybrid Studies

Basic scientific research continues to generate advances in treatment options, but bringing a product to market is not enough to ensure its optimal, appropriate, and safe use in clinical practice. In terms of improving health outcomes and use of resources, research to determine how, for whom, when, and in what context a treatment is best used deserves higher emphasis in funding and prioritization. As illustrated by John Rush of the University of Texas Southwestern Medical Center, a variety of methods are available for answering such complex and clinically important questions—such as using observational data obtained when systemic practices are employed (registry/cohort studies), effectiveness trials (practical clinical trials), and hybrid designs. Also of note are new study designs (e.g., equipoise stratified randomized designs; adaptive treatment studies) and posthoc data analyses (e.g., moderator analyses).

The results of a multisite NIMH trial, Sequenced Treatment Alternatives to Relieve Depression (STAR*D), illustrate how a hybrid trial can be employed to provide valuable insights on the optimal use of interventions in clinical practice as well as to improve future research analyses. This investigation helped to answer a host of clinically critical questions, such as the time needed to see a response in treatment; effects of sequence when a course of multiple treatment steps are involved; and effectiveness in a treatment population expanded beyond those in the efficacy trials to include patients with concurrent general medical conditions.

Expanding the evidence base to answer a broader set of questions important to delivering appropriate care—such as those examined in STAR*D—will require the design and use of cost-efficient, rapidly executed studies and the prioritization of research questions. Some key considerations include research that will change practice, enhance outcomes or understanding of the disorder, improve cost efficiency, and/or make treatments safer. Defining and answering these key questions will require the input of relevant stakeholders and care systems reengineered to support research. Finally, once the questions are defined, designs must be identified or developed to obtain the answers. NIH leadership will be essential, as will input and support from clinicians, patients, investigators, and payers. Without these commitments, how, when, for whom, and in what setting a treatment is best will remain the “art of medicine,” rather than the science.

Accommodating Genetic Variation as a Standard Feature of Clinical Research

Genomics is poised to have a significant impact on clinical care, and likewise, the medical system could, if adequately harnessed, dramatically transform our understanding of the role of genetic variation on disease development and progression. A significant threat to genomic medicine lies in its potential to generate a large number of spurious findings. To contend with this issue, the research community seeks approaches that garner a large number of patients needed to obtain reproducible linkages between disease characteristics and rare events or weak effects measured for common genetic variants. Given the potential offered by large-scale genomic studies, developing efficient and inexpensive approaches to obtain data of needed quality and quantity is of utmost importance. Issac S. Kohane of Harvard Medical School suggests three prongs of instrumentation of the health system in particular that will help to efficiently produce the large N needed: high-throughput genotyping, phenotyping, and sample acquisition.

Although the costs of high-throughput genotyping are rapidly dropping, the cost of assembling, phenotyping, and studying large populations is an estimated $3 billion for 500,000 individuals. Fortunately, the informational by-products of routine clinical care can be used to bring phenotyping and sample acquisition to the same high-throughput, commodity price point as is currently true of genotyping costs. Kohane discussed recent efforts to contend with challenges related to identifying relevant patient populations and obtaining biosamples from any phenotyped population. First, advances in automated natural language processing has allowed the evaluation of online health record text to quickly, reproducibly, and accurately stratify 96,000 out of 2.5 million patients for disease severity, pharmacoresponsiveness, and exposures. If expanded across many delivery systems, high-throughput phenotyping will be achievable at the national level. Second, a system currently being pilot-tested takes advantage of the many biosamples collected by laboratories in the course of care that are usually discarded. These samples present the opportunity to develop a potentially rich set of clinically relevant information (e.g., genomic composition, identification of biomarkers, effects of new treatments) that can be linked to previously phenotyped populations. A focus on instrumenting the health enterprise will not only contribute to advances in genomic studies but also provide an opportunity to learn from care delivery, if the appropriate security procedures are assured. The detection of cardiovascular risk in patients taking Vioxx was enabled by the development of large-scale databases and datamarts and is one example of how the healthcare system can be used for both discovery research and surveillance.

The most important step toward this goal is to create a safe harbor for methodological testing that challenges researchers to experiment with large datasets analysis. Open and transparent discussion is needed about the strengths and weaknesses of various methodological approaches; data should be made more broadly available so that researchers can test the data and methodologies and replicate findings. Progress will require increased investment in information technology—particularly to increase the quality of secondary uses of electronically captured data; addressing, through policy and education, various aspects of the Health Insurance Portability and Accountability Act (HIPAA) that prevent broader implementation of these systems and approaches; and the development of an informatics-savvy healthcare research workforce that understands relationships among health information, genomics, and biology.

Phased Introduction and Payment for Interventions Under Protocol

Clinical effectiveness research draws on experience gained in the post-market setting. Because this research is distinct from that required for FDA approval and market entry, innovative policies that encourage and facilitate these types of investigations will be needed. Wade Aubry from the Center for Medical Technology Policy described coverage with evidence development (CED) as an example of how policy can be used to contend with an essential problem in medical care—that for many clinical situations, evidence is insufficient to inform decision making. A brief history of how concepts evolved since CED’s initial use in the 1990s, as well as some lessons learned, is discussed.

An important early example of CED is of the support of commercial payers such as Blue Cross Blue Shield plans for patient care costs of high-priority, National Cancer Institute (NCI)-sponsored randomized clinical trials evaluating high-dose chemotherapy with autologous bone marrow transplantation. Other examples in Medicare include the FDA/Centers for Medicare & Medicaid Services (CMS) interagency agreement allowing for coverage of Category B investigational devices, coverage of lung volume reduction surgery for bullous emphysema under an NIH protocol, and the Medicare clinical trials policy, under which qualifying clinical trials receive Medicare coverage for patient care costs under an approved protocol. Over the past 4 years, Medicare CED has been formalized by CMS with guidance documents for CED policies on (1) implantable cardioverter defibrillators for prevention of sudden cardiac death, and (2) positron emission tomography for diagnosis of Alzheimer’s disease.

The application of CED to commercial health plans has grown in interest over the past year as part of the debate over whether a national comparative effectiveness institute should be established. Despite this progress, significant barriers to further development of this concept in the private sector remain. These include health plan Evidence of Coverage (EOC) language defining medical necessity, ethical issues, the difficulty in achieving multistakeholder consensus, lack of a clear definition of “adequate” evidence compared to “ideal” evidence, timing of CED in regard to existing coverage without restrictions, and limitations of the number of studies that can be implemented under CED.

Research Networks

Successful initiatives such as the Cooperative Oncology Groups, HMO Research Network, Center for Education and Research in Therapeutics, the Framingham Heart Study, and others provide useful models for how the clinical effectiveness paradigm might be redesigned to be more timely, relevant, efficient, and generalizable. As noted by Eric B. Larson from the Group Health Cooperative, similar emerging research networks embedded in the healthcare system provide an important anchoring element for a new generation of effectiveness studies. Certain characteristics of research networks make them particularly suited to produce good research on which to base clinical decisions.

Research conducted in a functioning delivery system is generally population based, with greater generalizability and relevance than research in convenience samples or highly selected, specialized populations. This research infrastructure also allows for better accounting for ecological factors in a real, live organization that can affect applicability of research. A well-constructed healthcare system not only enables more efficient conduct of trials and studies but also continuously pioneers technological and structural advancements to improve care. Such advancements are not the focus of traditional research but often significantly improve the quality of care.

Working in a functioning delivery system promotes another key characteristic: the “bidirectionality” of research. Research ideas often emerge from advances in more rarefied research settings when they may be ready for application to practice. Research ideas often surface because of real-world problems that need solutions or because innovative ideas emerge from the practice environment that need refinement and testing. Rather than thinking of only bench to bedside, the best research networks will perform bidirectional research and follow-up, so that there is, in essence, both an afferent and efferent limb to the research. Research networks are ideal for pragmatic and efficient clinical trials and, if population based, assessment of generalizability is ideal. That said, research networks placed in well-constructed healthcare systems will need to (and can) develop other, robust methods such as cluster randomized trials, disease registries, inception cohort studies and time series, and quasi-experimental use of observational data that can be helpful to a learning healthcare organization. Such research will also inform the field in general, especially if population based, allowing one to infer its generalizabilty. Research networks in well-characterized populations that are enrolled in an organized delivery system are ideal to reduce the time from research to translation, particularly in emerging research areas such as genomic/personalized medicine and use of electronic health records to efficiently determine phenotypes.

We are beginning to realize the potential of bringing together research networks in integrated healthcare systems with university-based scientists. One outcome of the NIH Roadmap has been development of NIH-funded programs under the Clinical and Translational Science Awards (CTSAs). The CTSA program aims to “develop a national system of interconnected clinical research networks capable of more quickly and efficiently mounting large-scale studies.” One consequence of this effort is a nascent culture change and, in places, work by institutions choosing to “reengineer” their clinical and translational research programs. A redesign of the clinical effectiveness research paradigm ideally would address challenges the NIH will face as it aims to reengineer the massive U.S. biomedical research enterprise. To achieve this culture change, both human and technical factors must be addressed: For CTSAs to work together, we will need a baseline level of interoperability to facilitate data exchange. It is also important to consider how CTSAs might seamlessly partner with other networks. Additionally, the research community needs to articulate and conscientiously attend to the challenges of building and sustaining strong collaborative teams.

Course-of-Care Data

Clinical systems increasingly capture data on the experience of each patient and clinician in a structured and quantifiable manner. These data resources present a major opportunity for “rapid learning” about the effectiveness of various treatments in the clinical practice setting. Greg Pawlson from the National Committee for Quality Assurance notes that although these data offer the opportunity to bridge the chasm between clinical practice and clinical or health services investigations, much work is needed to make these data available and useful for research. Key barriers to progress include insufficient funding for the development of the health information technology (HIT) infrastructure needed for rapid learning; the influence of HIPAA protections and the proprietary nature of these data on the research questions and approaches pursued; and the structure of the EHR.

Important to enabling the full potential of EHR and other course-of-care data resources will require greater focus on designing and developing EHR systems for research—beginning with the development of data standards and increased funding for the development of HIT infrastructure to facilitate access and use of data. Suggested policy interventions include:

• Better coordinating how the private and public sectors fund research, clinical learning, and HIT development;
• Increasing the proportion of funding dedicated to improving the quality and quantity of secondary database analyses;
• Creating incentives for collecting and structuring data useful for research;
• Providing more open and affordable access to data held by health plans and others;
• Engaging both the private and public sectors in an effort to set standards for how and what data are entered and retrieved from EHRs;
• Modifying HIPAA regulations to remove the major barriers imposed on research while balancing privacy concerns; and
• Improving medical education to better prepare health professionals to use individual and aggregated data for the care of patients.

Pharmaceutical Industry Data

As the healthcare system becomes more complex, the pharmaceutical industry is increasingly challenged to meet regulatory, payer, and patient demands for demonstration of the value of their products. Such assessments of risk–benefit, long-term safety, and comparative effectiveness often require postmarket clinical trial and database commitments. Peter Honig of Merck reflected on the difficult balance between the data transparency and data access needed to support the necessary epidemiologic, pharmaco-vigilance, and outcomes research with the increasingly commoditized and proprietary nature of data sources. Additional barriers to efficient use of these data include the decentralized nature of most utilization and claims outcome data and needed improvements in study and trial methods.

These issues are of particular interest to the industry in light of the high risk and high costs of the drug development process. Honig discusses several important initiatives underway to address these challenges. The FDA’s Critical Path initiative is advocating for public–private partnerships in the precompetitive space to address challenges in drug discovery and development, comparators other than placebo are being increasingly incorporated into clinical postapproval (Phase IV) trials, and structured methods are being developed to provide risk–benefit information that aids clinicians, payers, and regulators in making important decisions about safety, indicated use, and effectiveness. Industry acceptance of an ongoing, learning paradigm for evidence development has increased. Increasing rigor in pharmacoepidemiologic and risk–benefit standards is an asset to the field and the patient. The use of registries, both sentinel and population based, may provide a better method for pharmacovigilance. Increasing use of Bayesian statistical approaches, use of spontaneous and population-based data mining for postmarketing surveillance, and development of sophisticated data analysis tools to improve database output are all advances toward a smarter data collection system. To ensure these data add value to how care is delivered, educational efforts are also needed to improve the translation of generated knowledge into behavior.

Regulatory Requirements and Data Generation

Data developed and collected to satisfy regulatory requirements offer a rich resource and a driving force for improvements in our capacity for clinical effectiveness research. Mark B. McClellan of the Brookings Institution reports on two examples of how we might begin to take better advantage of these resources to enhance the healthcare system’s capacity to routinely generate and use data to learn what works in practice. The recently passed Food and Drug Administration Amendment Act of 2007 envisions the development of a postmarket surveillance system that actively monitors for suspected safety problems with medical products. However, it also has the potential to lead to the development of infrastructure built into the health-care system that can be used to address questions of effectiveness and use of products in different types of patients and populations. To take advantage of the opportunity presented, attention is needed to developing standards and consistent methods for defining adverse events and pooling relevant summary data from large-scale analyses, as well as contending with issues that impeded data sharing.

Medicare’s CED policy has also helped to develop the data needed to better inform coverage and clinical decisions by supporting the conduct of trials or the development of registries that collect and house sophisticated sets of clinical data about the use and impact of several medical technologies on patient outcomes. McClellan pointed out that clarification by Congress of CMS’s authority to use these methods is needed to support efforts currently underway and to encourage similar efforts taking place in the private sector.

These efforts seek to build the capacity to develop better evidence into the healthcare system, an approach that will become increasingly important to contend with the vast scope and scale of current and future knowledge gaps. The majority of these gaps are in areas for which it has been particularly challenging to develop evidence—such as assessing the effects of the many subtle and built-in differences in medical practice for patients with chronic disease. Because such information is critically important to improving outcomes, and will not be derived from traditional RCTs, priority efforts are needed to enhance the healthcare system’s capacity to generate data as a routine part of care and to use these data to learn what works in practice. Additional support is needed for the infrastructure, data aggregation, and analysis, and for improving the relevant statistical methods.

Ensuring Optimal Use of Data Generated by Public Investment

Though large amounts of data exist and have the potential to inform clinical and comparative effectiveness assessment, substantial barriers prevent optimal use of these data. Many innovative opportunities are possible from these publicly supported and generated data, such as the ability to inform clinical practice and policy. However, the restrictive interpretation of HIPAA and related privacy concerns, the growth of Medicare HMOs, and the fragmentation and commercialization of private-sector clinical databases all limit effectiveness research and threaten effectiveness findings. J. Sanford Schwartz referred to this as the paradox of available but inaccessible data and called for more attention to reducing the barriers to data use until they do not impede research effectiveness.

Enhanced coordination in the development of publicly generated data both within and across agencies can mitigate overlap and redundancy; the government should expand the RCT registry to include all comparative effectiveness research to further the range of issues addressed and information available. Access to data generated by public investment, including those by publicly funded investigators, should be expanded through the development of effective technical and support mechanisms. To move past the barriers presented by HIPAA, Medicare HMOs, and private-sector databases, Schwartz urged establishing practical, less burdensome policies for secondary data that protect patient confidentiality, expand Medicare claims files to incorporate new types and sources of data, and develop more cost-effective access to private-sector, secondary clinical data for publicly funded studies.

Building the Research Infrastructure

Given that evidence-based medicine requires integration of clinical expertise and research and depends on an infrastructure that includes human capital and organizational platforms, the NIH’s Alan M. Krensky said the NIH is committed to supporting a stable, sustainable scientific workforce. Continuity in the pipeline and the increasing age at which new investigators obtain independent funding are the major threats to a stable workforce. To address these concerns, the NIH is developing new programs that target first-time R01-equivalent awardees with programs such as the Pathway to Independence and NIH Director’s New Innovator Awards, with more than 1,600 new R01 investigators funded in 2007. NIH-based organizational platforms are intra-and interinstitutional. CTSAs fund academic health centers to create homes for clinical and translational science, from informatics to trial design, regulatory support, education, and community involvement. The NIH is in the midst of building a national consortium of CTSAs that will serve as a platform for transforming how clinical and translational research is conducted. The Immune Tolerance Network (ITN), funded by the National Institute of Allergy and Infectious Diseases, the National Institute of Diabetes and Digestive and Kidney Diseases, and the Juvenile Diabetes Research Foundation, is an international collaboration focused on critical path research from translation to clinical development. The ITN conducts scientific review, clinical trials planning and implementation, tolerance assays, data analysis, and identification of biomarkers, while also providing scientific support in informatics, trial management, and communications. Centralization, standardization, and the development of industry partnerships allow extensive data mining and specimen collection. Most recently, the Immune Tolerance Institute, a nonprofit, was created at the intersection of academia and industry to quickly transform scientific discoveries into marketable therapeutics. Policies aimed at building a sustainable research infrastructure are central to support evidence-based medicine.

Engaging Consumers

Conducting meaningful clinical effectiveness research requires collecting, sharing, and analyzing large quantities of health information from many individuals, potentially for long periods of time. To be successful, this research will need the support and active participation of patients. The relationship between researcher and research participant, as defined by current practice, is ill suited to successfully leverage such active participation. As reported by Kathy Hudson of Johns Hopkins University, however, public engagement efforts in biomedical research, while still in their infancy, suggest some key challenges and opportunities for cultivating active public participation in clinical effectiveness research.

The biomedical community—and the science and technology community more generally—traditionally have viewed the linear progression from public education to public understanding to public support as an accurate model through which to cultivate a public enthusiastically supportive of and involved in research. As the flaws in this philosophy have become more apparent, research-performing institutions increasingly are turning to public engagement and public consultation approaches to enlist public support. Unlike unidirectional and hierarchal communications that characterize past efforts, public engagement involves symmetric flow of information using transparent processes and often results in demonstrable shifts in attitudes among participants (though not always in the direction one might expect or prefer). The outcome is different, as well: Rather than aspiring for or insisting on the public’s deeper understanding of science, a primary goal of public engagement is the scientists’ deeper understanding of the public’s preferences and values.