2.1. Statement of Objective

Most productive epidemiologic research begins with a clear statement of objective. This objective might be descriptive; for example, to measure the incidence of a particular disease in some population, to characterize the patterns or costs of treatment for a particular disease in some population, or to measure the occurrence of outcomes among patients with a particular disease.

The objective might also involve a comparison; for example, to compare the incidence of a particular disease in two or more subgroups defined by common characteristics (e.g., etiologic research), to compare the cost or quality of care for a particular disease in two or more subgroups (e.g., health services research or disparities research), or to compare the rate of outcomes among two or more subgroups of patients (often defined by different types or levels of treatment) with a particular disease (e.g., clinical research). In all cases, the overarching objective is to obtain an accurate (valid and precise) and generalizable estimate of the frequency of an outcome's occurrence, or its relative frequency compared across groups.¹⁰ A valid estimate is one that can be interpreted with little influence by systematic error. A precise estimate is one that can be interpreted with little influence by random error. A generalizable estimate is one that provides information pertinent to the target population, the population for which the study's information provides a basis for potential action, such as a public health or medical intervention. Often times the objective will be accompanied by a specific hypothesis (see Chapter 13), although that is less important than the statement of objective.

2.2. Selection of a Study Population

Once the study's objectives have been stated, the next step in the research plan is to select a study population. Selection of a study population requires identifying potential participants in time and place, including inclusion/exclusion (admissibility) criteria related to the study's objectives and feasibility. Admissibility criteria related to the study's objectives include focusing on a clinically relevant study population of individuals in whom sufficient events will occur to provide adequate precision for the estimates of disease frequency, and in whom the exposure categories will occur with sufficient frequency to provide adequate precision for the estimates of association. These criteria are also used to exclude people with characteristics that can introduce significant bias into the estimates of disease frequency or estimates of association, and that cannot be controlled easily or adequately in the analysis. Precision and validity criteria for admissibility pertain to all studies, regardless of whether they are nested in a health database.

Admissibility criteria related to feasibility center on access to the data. Many ongoing cohort studies have established procedures for data sharing. Similarly, most publicly funded health databases have established procedures for data access. Investigators must ordinarily provide a statement of the study's objective, a protocol for data collection from the database and for data analysis, a list of individuals who will have access to the data, and a study timeline. Some databases charge a fee for data access, although many do not.

An advantage of retrospective database studies is the potential to study associations between rare exposures and rare outcomes in a population large enough to provide sufficient precision, with nearly complete followup, and with few exclusion criteria pertaining to age, comorbidity, or other factors that sometimes limit participation in clinical trials.^{11, 12} For example, surveillance databases that monitor adverse events potentially associated with pharmaceuticals identified signals suggesting that use of HMG CoA-reductase inhibitors (statins) might increase the risk of amyotrophic lateral sclerosis (ALS).^{13, 14} The only available epidemiologic evidence came from pooling 41 randomized trials, in which ten ALS cases occurred among 56,352 individuals assigned to placebo and nine ALS cases occurred among 64,602 individuals assigned to the statins arm.¹⁴ Using Danish databases, a case-control study identified 556 cases of ALS or other motor neuron syndromes and 5,560 population controls.¹⁵ The odds ratio associating disease occurrence with statins use was 0.96 (95% CI, 0.73 to 1.28), thereby rapidly and cost-efficiently providing evidence to counter the drug-monitoring studies and with far greater precision than provided by the pooled clinical trials.

Selection of a study population inevitably involves balancing accuracy and generalizability concerns, as well as cost and feasibility considerations. For example, restriction is one of the most effective strategies for control of confounding through study design.¹⁶ If one is concerned about confounding by sex, a simple and effective strategy to control that confounding is to restrict the study population to a single sex. However, such restriction reduces the study's precision by decreasing the sample size, and may also reduce the generalizability of the results (only applicable to half of the target population). An alternative would be to include both sexes and to stratify the analysis by sex. While this approach would improve the generalizability of the results, and allow an evaluation of confounding, the precision of the estimated association would be reduced, and perhaps substantially reduced, if the estimate of effect in men was substantially different from the estimate of effect in women. In this circumstance, the study becomes effectively two studies.

2.3. Definition of Analytic Variables

The protocol for an epidemiological study should provide a clear, unambiguous definition of the outcome being studied, a description of how it will be measured, and a discussion of the accuracy of that measurement. When sensitivity of a dichotomous disease classification is nondifferential and independent of any errors in classification of exposure categories, and there are expected to be few false positives (near perfect specificity), there will usually be little bias of a ratio measure of association.⁴ This exception to the rule that “nondifferential misclassification biases towards the null” has important design implications. It suggests that retrospective database studies should be designed to optimize specificity; in fact, to ideally make the specificity perfect so there will be no false positives. Such a design might require more stringent criteria applied to the outcome definition than are ordinarily applied in a clinical setting, and therefore more stringent than might be found in a disease registry. For example, the estimated prevalence of dementia in a cohort of men and women aged 65 years or older varied by a factor of 10 depending on the diagnostic criteria that were applied.¹⁷ Strategies to reduce inclusion of false-positive cases can include requiring evidence in the patient record of medical procedures (e.g., cholecystectomy for gallstone disease or podiatry examination for diabetes) or interventions (e.g., insulin or glucose lowering medications for diabetes) that provide greater confidence in the validity of the case-finding definition.¹⁸ Such an approach often results in fewer included cases and reduced precision, but improved validity.¹⁹

If the study objective is to compare the frequency of outcome across subgroups, then the protocol should provide a definition of the exposure contrast(s). It is critical that both the index condition (i.e., the “exposed” or “treated” group) and the reference condition (i.e., the “unexposed” or “untreated/placebo” group) are well defined.^{6, 20} One frequent shortcoming of epidemiologic research is to compare the occurrence of disease in an index group with the occurrence of disease in all others who do not satisfy the index group definition. Studies of this design are easily constructed with retrospective database research, because of the abundance of participants who do not meet the index group definition. This “all others” reference group is therefore usually a poorly defined mixture of individuals.²¹ For example, if one uses a pharmaceutical registry to compare the incidence of a disease in statins users with the incidence of disease in those who do not use statins, the reference group of nonusers will contain individuals with indications for statin use but who have not been prescribed statins, as well as individuals without indications for statins use. Nonusers also differ from users in the frequency of contact with medical providers, which raises the potential for differential accuracy of ascertainment of health outcomes. It is therefore preferable to first ensure that the reference group of nonusers contains individuals who have indications for use of the treatment,¹⁸ and who, if possible, are receiving alternative therapies for the same indication.²² If one has a biologic basis to separate statins into categories, such as hydrophilic and hydrophobic statins, then a comparison of users of hydrophilic statins with users of hydrophobic statins would often be more valid. With these definitions, only individuals with indications for statins, and treated with statins, are included in the analysis, thereby reducing the potential for confounding by indication and differential followup.²³

Finally, considerable attention should be given to identifying and accurately measuring potential confounders and effect modifiers.^{4, 24} The opportunity to examine important etiologic questions with considerable precision has expanded significantly with the availability of large databases, but systematic error due to confounding by unmeasured or poorly measured confounders remains a central concern. Fortunately, databases generally capture inpatient and outpatient clinical events and medication use that can characterize comorbidities and health care resource utilization, which can aid in the control of confounding. As discussed further below, information on behavioral and lifestyle factors (e.g., cigarette smoking, alcohol use, diet) is infrequently captured or is poorly measured in many databases. Thus, retrospective database researchers should carefully consider the available information on confounders before initiating studies. When data on critical confounders cannot be obtained in a database, and cannot be obtained by linking to another data source, an alternative data set might be better suited to accomplish the study's objectives. Alternatively, in the presence of unmeasured confounding, researchers can use bias analysis^{5, 25} to assess the potential impact of residual confounding on their observed findings.^{26, 27}

2.4. Validation Substudies

The goal of quality study design and analysis is to reduce the amount of error in an estimate of association. With this goal in mind, investigators have an obligation to quantify how far they are from this goal, and bias analysis is one method by which to achieve this goal.^{5, 25} Bias analysis methods require data to inform the bias model, and these data are obtained from internal or external validation substudies. Retrospective database research is often amenable to collection of internal validation data, for example by medical record review. In addition, many databases have internal protocols that constantly validate at least some aspects of the data. The validation data generated by these protocols can provide an initial indication of the data quality. To facilitate data collection for study-specific internal validation studies, investigators should consider the important threats to the validity of their research while designing their study, and should allocate project resources accordingly. This consideration should immediately suggest the corresponding bias analyses, which will then inform the data collection required to complete the bias modeling.

For example, in the study of statin use related to ALS and neurodegenerative diseases described above,¹⁵ the ICD-10 code used to identify cases (G12.2) corresponded to diagnoses of ALS or other motor neuron syndromes. The investigators therefore selected a random sample of 25 individuals from among all those who satisfied the case definition, and a clinician investigator reviewed their discharge summaries. The proportion of these 25 who did not have ALS (32 percent) was used to inform a bias analysis to model the impact of these false-positive ALS diagnoses. Assuming a valid bias model, the bias analysis results showed that the null association was unlikely to result from the nondifferential misclassification of other diseases as ALS.

In this example, there was no effort to validate that non-cases of ALS were truly free of the disease. Non-cases are seldom validated, because false-negative cases, especially of rare diseases, occur very rarely. Furthermore, validating the absence of disease often requires a study-supported medical examination of the non-case patients, an expensive, time-consuming, and invasive procedure. Prevalent diseases with a lengthy preclinical period and relatively simple diagnostic tests, such as diabetes, are more amenable to validation of non-cases. The ALS example also illustrates that an internal validation study requires protocol planning and allocation of study resources to collect the validation data. A protocol should be written that specifies how participants in the validation sample will be selected from the study population. Participation in the validation substudy might require informed consent to allow medical record review, whereas the database data itself might be available without individual informed consent. These aspects should be resolved in the planning stage, and the analytic plan should include a section devoted to bias modeling and analysis.⁵

3.1. Structural Framework for Data Collection

Health databases collect data for various primary purposes³⁰ and can be categorized as follows: (1) data collected for the purpose of reimbursing health care providers; (2) data collected for the purpose of monitoring care provided to beneficiaries of an integrated health care system; (3) data collected for the purpose of surveillance regarding a particular disease or disease category; (4) data collected for the purpose of surveillance for individuals with a specific exposure; and (5) data collected on individuals with a single admission-defining disease or medical procedure. Each type has strengths and limitations (presented in Table 18–1) to consider when evaluating the database for use in studies.

Table 18–1

Types of databases used for retrospective database studies, and their typical advantages and disadvantages.

Databases that collect information for reimbursement (e.g., Medicare, Medicaid, or Ingenix), which are sometimes called “claims” or “administrative” databases, are quite useful for understanding health care costs and can provide important surveillance information on clinical practices and outcomes. However, they may be susceptible to systematic errors if data entries are manipulated by the data generators to affect (likely increase) their reimbursement. For instance, certain clinical conditions with high reimbursement rates may be preferentially reported on claims for patients who have those conditions but who present in the hospital or outpatient setting with other clinical issues, particularly if the presenting conditions are reimbursed at lower rates. The accuracy of some claims data sets have been questioned for diagnoses and procedures including dialysis,³¹ weight management,³² neutropenia,³³ heart failure,³⁴ diabetes,³⁵ and functional outcomes after prostatectomy,³⁶ as examples. On the other hand, the accuracy of registered diagnoses can be quite good.³⁷ The accuracy of the claims data for its intended objective should therefore be considered, and preferably estimated quantitatively by an internal validation substudy.^{38, 39} Alternatively, estimates of the data's accuracy may be available from an analogous study population from the same or a similar claims data set; an example is an external validation study. Claims data often lack important information on laboratory parameters, diagnostic test results, and behavioral and lifestyle characteristics, which may limit their utility for research in some topic areas.

The second type of database collects information on the health care provided to beneficiaries within an integrated health care system. This system can be a health insurer (e.g., Kaiser Permanente), a benefits program provided to selected individuals (e.g., Veteran's Health Administration), or a national health care system (e.g., the United Kingdom's Clinical Practice Research Database⁴⁰). These databases typically use an integrated electronic health records system to capture health care information directly from physicians' offices, hospitals, pharmacies, and other sites where care is provided (e.g., infusion centers, surgical centers). The granularity and quality of data captured in these databases is quite good and includes demographic and clinical characteristics, medication use, major clinical events including death, and importantly, results of diagnostic tests and laboratory assays. As with many epidemiological studies, some databases are limited in their geographic coverage and in the demographic characteristics of their patient populations. This lack of representativeness may affect the generalizability of results from studies nested in them.

The intended purpose of a third set of databases is surveillance of the incidence and outcomes related to a particular disease or disease category. These databases, or surveillance registries, often pertain to infectious diseases, cancer, and end-stage renal disease (ESRD). Surveillance for infectious diseases sometimes recognizes that only a proportion of cases will be reported, but assumes that the sensitivity and specificity of reporting remain constant over time, so that changes in the relative frequency of reported incidence provides a signal regarding the true incidence in the population. Thus, although the data quality is high, the completeness may be low. In contrast, both the data quality and completeness in most cancer registries are quite high, and the motivation for manipulation to influence reimbursement does not exist because the registry data are not used for that purpose. For example, the U.S. Cancer Surveillance, Epidemiology, and End Results (SEER) registry has a history of quality control and improvement dating to its inception in 1973 and has been linked to the Medicare administrative database to provide data on cancer treatments and outcomes. In the United States and some other countries, patients with ESRD (patients receiving chronic dialysis or who are transplant recipients) are guaranteed coverage of all dialysis services including medications, procedures, and hospitalizations. These benefits extend throughout the patient's life and require significant resources. Consequently such countries have established surveillance programs like the United States Renal Data System to monitor the health care provided to these patients and the costs associated with their health care.

The fourth type of database collects data on patients with a common exposure, and is commonly used as part of a postmarketing pharmacovigilance program related to a biologic or pharmaceutical product or a medical device. This type of database is typically designed to monitor the incidence of adverse events related to the exposure. These databases are often patient registries.

A last type of database is a clinical patient registry of individuals with a single admission-defining disease or medical procedure. In fact, the first known health-related registry was the Leprosy Registry in Norway, initiated in 1856. In keeping with this history, many of the current clinical registries are found in Scandinavia. For example, the Danish government supports clinical databases used for quality assurance and research (e.g., breast cancer, colorectal cancer, hip arthroplasty, and rheumatologic diseases), as well as disease registries (e.g., the multiple sclerosis registry) used for monitoring and research.⁴¹ In fact, a central objective of disease-specific registries may be to provide an infrastructure for clinical trials pertaining to treatments for the disease. The main advantage of these registries and databases is the quality of data on disease characteristics, received treatments, and outcomes related to the disease. The main disadvantage is that they are difficult to use for studies of the etiology of the disease that initiates membership in the registry, since the registry includes only individuals with the disease.

3.2. Changes in Coding Conventions Over Time

A common problem with retrospective database research is the impact of changes in coding conventions over the lifetime of the database. These changes can take the form of diagnostic drift,⁴² changes in discharge coding schemes, changes in the definition of grading of disease severity, or even variations in the medications on formulary in one region but not others at different points in time. For example, the Danish National Registry of Patients (DNRP) is a database of patient contacts at Danish hospitals. From 1977 to 1993, discharge diagnoses were coded according to ICD-8, and from 1994 forward discharge diagnoses were coded according to ICD-10. ICD-10 included a specific code for chronic obstructive pulmonary disease (J44), whereas ICD-8 did not [ICD-8 496 (COPD not otherwise specified) did not appear in the DNRP]. In addition, from 1977 to 1994 the DNRP registered discharge diagnoses for only inpatient admissions, but from 1995 forward discharge diagnoses from outpatient admissions and emergency room contacts were also registered. COPD patients seen in outpatient settings before 1995 were therefore not registered; this excluded patients who likely had less severe COPD on average. The change in ICD coding convention in 1994 and the exclusion of outpatient admissions before 1995 presented a barrier to estimating the time trend for incidence of all admissions for COPD in any period that overlapped these two changes to the DNRP.⁴³

The General Practice Research Database (GPRD) was a medical records database capturing information on approximately 5 percent of patients in the United Kingdom⁴⁴ (as of March 2012, the GPRD became the Clinical Practice Research Database). Information was directly entered into the database by general practitioners trained in standardized data entry. When the GPRD was initiated in 1987, diagnoses were recorded using Oxford Medical Information Systems (OXMIS) codes, which were similar to ICD-9 codes. In 1995, the GPRD adopted the Read coding system, a more detailed and comprehensive system that groups and defines illnesses using a hierarchical system. Without knowledge of this shift in coding and how to align codes for specific conditions across the different coding schemes, studies using multiple years of data could produce spurious findings.

3.3. Other Data Quality Considerations

3.4. Confounding by Indication

Confounding by indication may occur in nonrandomized epidemiologic research that compares two treatments (or treatment with no treatment).⁵⁸ In the absence of randomization, the indications for selecting one treatment in preference to another (or in preference to no treatment) are often also related to the outcome meant to be achieved or prevented by the treatment.⁵⁹ For example, randomized trials in younger breast cancer patients have shown that chemotherapy prevents breast cancer recurrence.⁶⁰ However, in a nonrandomized study of older breast cancer patients, those who received chemotherapy had a higher rate of recurrence than those who did not, probably because chemotherapy was offered only to the women with the most aggressive cancers.⁶¹ This example is a classic illustration of confounding by indication. Importantly, this study collected complete detailed data on every prognostic marker of recurrence and all of the other breast cancer treatments, yet adjustment for this detailed suite of variables did not resolve the confounding by indication, even using more advanced methods.²³

Retrospective database research is as susceptible to confounding by indication as any other design. However, strategies to reduce the strength of this confounding have been proposed²¹ and may be most successful when used in the large study populations often achievable only in databases.⁶² Explained here is a special class of confounding by indication, which might arise especially in retrospective database research: time-dependent confounding by indication generated by dynamic dosing.⁶³ Dynamic dosing refers to the clinical situation in which a medication's dose is titrated (increased or decreased) in response to a changing biomarker or clinical measurement on which the medication acts (i.e., a clinical intermediate).⁶³ Examples include diabetes medications titrated in reaction to hemoglobin A1c (HbA1c) measurements, erythropoiesis stimulating agents (ESAs) titrated in reaction to hemoglobin levels, blood pressure medications titrated in reaction to systolic and diastolic blood pressure values, and antiretroviral therapy titrated in reaction to CD4 counts. The clinical intermediate is therefore both a consequence of therapy and a predictor of future therapy. Time-dependent confounding arises when the clinical intermediate is also a prognostic indicator.⁶⁴ For example, hemoglobin concentration is a time-dependent confounder of the effect of ESA therapy on survival because it is a risk factor for mortality, it predicts future ESA dose, and past ESA therapy predicts future hemoglobin concentration. Dynamic dosing therefore introduces time-dependent confounding of the treatment's association with outcomes in the presence of this structure of confounding by indication.⁶³

It is important to recognize that the structure requires the clinical intermediate to be both a causal intermediate and a confounder. If it is only a confounder, such as baseline comorbidity or time-dependent comorbidity, the confounding can be addressed by conventional analytic methods. However, when the causal structure indicates that the clinical intermediate is both a causal intermediate and a confounder, inverse probability of treatment weighting (IPTW) with marginal structural models (MSMs) has been proposed as one method for valid adjustment.⁶⁵ Pharmacoepidemiological studies that have used MSMs to address time-dependent confounding have shown significant improvements in confounding control relative to traditional time-dependent analysis.^66-68 In a study of the effect of highly active antiretroviral therapy (HAART) on time to AIDS, the hazard ratio using standard time-dependent Cox regression to adjust for time-varying covariates such as CD4 count and HIV RNA level was 0.81 (95% CI, 0.61 to 1.07). Using an MSM, this effect was strengthened substantially (HR=0.54, 95% CI, 0.38 to 0.78), providing stronger evidence of the benefit of HAART.⁶⁶ Studies examining the effect of titrated ESA doses on mortality risk in dialysis patients that have used MSMs have found hazard ratio estimates at or below the null,^{67, 68} whereas results from traditional models found substantially elevated hazard ratio estimates.⁶⁷

3.5. Precision Considerations When Standard Errors Are Small (Over-Powered)

The large size of the study population that can often be included in retrospective database study is both a strength and a limitation. The sample size allows adjustment for multiple potential confounders with little potential for over-fitting or sparse data bias,⁶⁹ and allows design features such as comparisons of different treatments for the same indication (comparative effectiveness research) to reduce the potential for confounding by indication.²¹ Nonetheless, systematic errors remain a possibility, and these systematic errors dominate the uncertainty when estimates of association are measured with high precision as a consequence of a large sample size.⁷⁰ When confidence intervals are narrow, systematic errors remain, and/or inference or policy action will potentially result, investigators have been encouraged to employ quantitative bias analysis to more fully characterize the total uncertainty.²⁵ Bias analysis methods have been used to address unmeasured confounding,²⁷ selection bias,⁷¹ and information bias^{27, 72} in retrospective database research.

A second potential problem is the possibility of overweighting results from retrospective database–based research in a quantitative meta-analysis of an entire body of research on a particular topic. In such meta-analyses, weights are in proportion to the inverse of variance, so large studies carry most of the weight. The variance, however, measures only sampling error; it does not measure systematic error. This problem of large studies dominating the weights pertains to any meta-analysis that includes one or two studies much larger than the others. However, given the large sample sizes often achieved by retrospective database research, the high-weight studies may often come from studies nested in these databases. For example, in a 2004 quantitative meta-analysis of 11 prospective studies of the association between pregnancy termination and incident breast cancer,⁷³ the two retrospective database studies^{74, 75} accounted for 54 percent of the weight in the meta-analysis, but only 18 percent (2 of 11) of the studies. Random effects meta-analyses⁷⁶ and other weighting methods⁷⁷ provide only a partial solution to this potential overweighting, and only in some circumstances. Meta-analysts should therefore consider the potential for retrospective database research to be overweighted in their quantitative summary estimates. A plot of the inverse-normal of rank percentile against the corresponding study's estimate of association and confidence interval provides a visual depiction of the distribution of study results,⁷⁸ without undue influence by overpowered studies. (See, for example, the aforementioned meta-analysis of the association between pregnancy termination and breast cancer risk.⁷³)

4.1. Rapid Response to Emerging Problems, With Prospective Data

Retrospective database research is ordinarily secondary to another primary purpose. While the collected data may not be optimized to a particular research topic, it is often possible to use the collected data for rapid response to emerging research problems.⁷⁹ The study mentioned above of the association between statins medication and incident ALS is also a suitable example here. Drug surveillance databases had identified a higher-than-expected prevalence of statins medications associated with reports of ALS. A pooled analysis of trials data revealed no association, but was limited by the small number of ALS cases, short duration of followup, and potential for crossover from the placebo arm to statins treatment after the trial finished.⁸⁰ Thus, there was little evidence to evaluate the potential causal association between this highly effective drug class—which prevents cardiovascular morbidity and mortality^{81, 82}—and the incidence of ALS, a progressive, neurodegenerative, terminal disease.⁸³

The precisely measured null association reported in the case-control study15 provided a rapid and reliable basis to assuage concerns about an etiologic association between statin use and ALS occurrence. Imagine what would have been required for a purposefully designed study to evaluate the association. The pooled trials result had included nearly 120,000 individuals observed over more than 400,000 person-years, yet included only 19 cases of this rare disease. Few existing cohort studies would have had sufficient person-time to expect substantially more cases, and a cohort study designed to evaluate the association would have required a substantial investment of time and financial support.

A case-control study might have been feasible, but imagine the resources required to enroll and interview an equivalent number of ALS cases as were included in the database study (∼550) and their matched controls. Furthermore, a case-control study of this design would likely have been susceptible to recall bias and selection bias.^{4, 7} The retrospective database research study avoided both of these biases.¹⁵ Recall bias was avoided by ascertaining statins use from a prescription database. These prescriptions were recorded before the ALS incidence, so could not have been affected by the subsequent disease occurrence. Selection bias was avoided because all ALS cases in the region during the followup period were included, and controls were selected from the Civil Registration System. Neither case/control status nor use of statins was likely to be associated with participation. Thus, the retrospective database research study on this topic provided a rapid, cost-efficient, and precise result on an important public health topic, which otherwise would have gone unevaluated or would have required a substantial investment of time and finances to achieve an equivalent, or possibly more biased, result. This study provides a good example of the value of retrospective database research in such circumstances.

4.2. Cost-Efficient Hypotheses-Scanning Analyses

Retrospective database research can sometimes evaluate multiple associations with only a marginal increase in cost over the evaluation of a single association. The U.S. Food and Drug Administration's (FDA) Sentinel Initiative will use an active surveillance system within electronic data from health care information holders to monitor the safety of all FDA-regulated products.⁸⁴ Similarly, the EU-ADR project aims to use clinical data from health databases, combined with prescription databases, to detect adverse drug reactions.^{85, 86} The project uses text mining, epidemiological, and computational techniques to analyze electronic health records, with the goal of detecting combinations of drugs and adverse events that merit further investigation.

As a second example, Latourelle and colleagues used retrospective database research to evaluate the association between estrogen-related diseases, such as osteoporosis or endometriosis, and the occurrence of Parkinson's disease.⁸⁷ To be categorized as “exposed” to these diseases, cases or controls had to have them appear as discharge codes in the hospital database before the first discharge code for Parkinson's disease. For relatively little additional cost, the investigators also evaluated the association between 200 other diseases and the subsequent diagnosis of Parkinson's disease as a hypothesis scanning study, with the objective of suggesting new ideas regarding Parkinson's disease etiology.⁸⁷ The analysis adjusted for multiple comparisons using empirical Bayesian methods designed to reduce the emphasis on potentially false-positive associations.⁸⁸ This potential for cost-effective hypotheses-scanning studies as an explicit objective of retrospective database research should be viewed as a strength of such research, not a limitation, so long as the objective is appropriately labeled as such. Hypotheses suggested by these types of studies are often further investigated using studies designed specifically for the topic.

4.3. Hybrid Designs

Retrospective database research does not necessarily have to be limited to data collection from secondary data sources. Hybrid designs allow the use of database research for some aspects of data collection, and primary data collection for others. For example, a study of drug-drug and gene-drug interactions that might reduce the effectiveness of tamoxifen therapy began by identifying eligible breast cancer patients using the Danish Breast Cancer Cooperative Group's clinical registry.⁸⁹ This clinical registry also provided data on prognostic factors such as tumor diameter and lymph node evaluation, and on treatments such as chemotherapy and radiation therapy. Linkage with the Danish Civil Registration System provided data on vital status; linkage with the Danish National Patient Registry provided data on comorbid diseases; and linkage with the Danish National Registry of Medicinal Products provided data on use of prescription medications. Thus, for relatively low cost, a cohort of breast cancer patients with complete medical, prognostic, and breast cancer treatment data was assembled. A case-control study was then nested in this cohort by identifying cases of breast cancer recurrence and then matching controls to them by risk-set sampling.⁷ Once cases and controls had been identified, their tumor blocks were collected from the Danish National Pathology Registry,⁹⁰ and these were used for the necessary bioassays. Thus, retrospective database research allowed identification of the source population and selection of cases and controls, and provided all but the bioassay data. These data, which are expensive to collect, were only obtained for about 13 percent of the members of the total cohort. This hybrid design demonstrates that retrospective database research will remain an important contributor, even in the era of personalized medicine.

In a second example of a hybrid design, survey data collected over the Internet were linked to retrospective database research.^{91, 92} The objectives of the study were to assess the feasibility and validity of studies that use the Internet to recruit and follow participants, evaluate the relationship between lifestyle and behavioral factors and delayed time to pregnancy among women attempting to conceive, and evaluate the relationship of several exposures to risk of miscarriage and infant birth weight among women who conceived. Participants were recruited by advertisements on Web sites likely to be visited by women who intended to become pregnant. They were directed to the study's Web site, where they completed an enrollment screening questionnaire followed by an interview covering socio-demographics, reproductive and medical history, lifestyle, and other factors. Enrolled participants were then contacted every 2 months by email for 12 months or until they reported that conception had occurred. Data obtained from the Web-based questionnaires were linked to nationwide databases, which allowed collection of additional data on confounders and outcomes, as well as an assessment of the validity of some of the self-reported data, such as prescription drug use. This study again demonstrated that retrospective database research, in combination with primary data collection, can provide a cost-efficient resource for collecting some aspects of the study data. In contrast to the previous cancer treatment example, the cohort in this pregnancy study was enrolled following more typical cohort study strategies, and not by using the databases to identify a source population.

Hybrid designs have also been used to collect data by medical record review for data fields that are available for a subset of participants in a database.⁹³ Thus, the database provides a cost-efficient resource for initial data collection, which is then supplemented as necessary by medical record review or another primary data collection method to complete the data set. Once an investigator is open to the potential for hybrid designs and there are retrospective database resources suitable to the research topic, the opportunities for combining the databases with primary data collection are limited only by the investigator's creativity.

4.4. Ample Data Allows for Novel Designs

As mentioned above, the ample data often available from retrospective database research can lead to overweighting of such studies in quantitative meta-analyses. While this problem may be disadvantageous, a compensating advantage is the opportunity to use retrospective database research to implement novel study designs. For example, confounding by indication and other biases often plague clinical epidemiology,^{3, 23} even in the era of comparative effectiveness research. However, the ample study size often provided by retrospective database research can overcome these threats to validity in some situations. The large sample size might allow a design with carefully restricted exposure groups,1 for example, new users of a pharmaceutical only,⁹⁴ whereas conventionally sized cohort studies would not always have sufficient study size to implement such a design. The new user design in turn facilitates other advanced designs, such as propensity score matching and instrumental variable analyses,²¹ which are intended to further counteract these threats to validity. These and other novel designs can be implemented in studies of any size, but are likely most effective when the study size is large.⁹⁵

4.5. Data Pooling Methods

Although retrospective database research often provides relatively large study size within a research topic area, a study's power may still be insufficient if the study must be restricted to rare exposure subgroups or if the study outcome is rare. In these cases, data pooling across similar databases may allow sufficient sample size to provide adequate power. Data pooling also provides advantages over conventional meta-analyses because it allows simultaneous and consistent data analyses. However, such pooling projects face substantial challenges.

First among these challenges is harmonization of the data elements. To accommodate a pooled analysis, data collected from different databases must provide analytic variables (exposure, confounders, modifiers, and outcomes) with equivalent categorizations and definitions. Such data harmonization can be quite challenging. Harmonization of data elements categorized differently or differentially available in two or more databases may pose an insurmountable barrier to pooling. For example, one database might include data on behaviors like alcohol and tobacco use, whereas a second database might not. The pooling project would then face the unenviable decision of controlling for these behaviors for some, but not all, data centers (in which case the analysis becomes comparable to a conventional meta-analysis), or abandoning control for these variables at all centers in order to achieve the data harmonization goal. Differences in the conceptual underpinnings of data elements may be more common. Even a variable as conceptually simple as the Charlson comorbidity index⁹⁶ can present surprising challenges when subject to harmonization considerations. The Charlson index includes 19 comorbid conditions (e.g., diabetes). As mentioned above, some databases might be able to ascertain diabetes diagnosed in all medical settings (e.g., general practitioner, outpatient, and inpatient), whereas others might be able to ascertain diabetes diagnosed in only a subset (e.g., only general practitioner or only outpatient specialty clinics). Diabetes is defined differently in the different databases, which are not strictly harmonious, and therefore contribute differently to the Charlson index. While the definition of the Charlson variable may be harmonious across the pooled databases, the underlying conceptualization is different, and this difference could result in differences in the strength of confounding by the comorbidity variable or in the degree to which it modifies the association between an exposure contrast and outcome.

Ethical and legal constraints, which are often placed on data sharing, present a second important challenge to pooling projects. Pooling of de-identified data sets can sometimes be arranged through data use agreements, but even these arrangements can be quite challenging and time-consuming. Rassen and colleagues compared four methods of pooling de-identified data sets:⁹⁷ (1) full covariate information, which may violate privacy concerns; (2) aggregated data methods, which aggregate patients into mutually stratified cells with common characteristics, but usually delete cells with low frequency counts that might defeat the privacy protections of large frequency counts; (3) conventional fixed or random effects meta-analysis, which provides only summary estimates of association for pooling; and (4) propensity score-based pooling, for which a propensity score summarizes each individual's covariate information. They reported that the last alternative provided reasonable analytic flexibility and also strong protection of patient privacy, and advocated its use for studies that require pooling of databases, multivariate adjustment, and privacy protection.⁹⁷

More recently, Wolfson and colleagues proposed a pooling method that requires no transfer of record-level data to a central analysis center.⁹⁸ Rather, the central analysis center implements statistical computing code over a secure network, accessing record-level data maintained on servers at the individual study centers. Data aggregation occurs through return of anonymous summary statistics from these harmonized individual-level databases, and even iterative regression modeling can be implemented. The advantage is a reduced burden to comply with ethical and legal requirements to protect privacy, since no record-level data are ever transferred. The disadvantages include requirements for strong data harmonization, secure networks that satisfy regulatory oversight, and assurances that no record-level data are transmitted. It is possible that some summary statistics could violate standards for de-identification, but safeguards can be implemented to prevent transmission of such summary statistics.

These new methods for pooling provide exciting opportunities for pooled projects. At the time of this writing, investigators who choose to undertake them should expect delays required to explain these methods to regulators responsible for oversight of data protection, who are not yet familiar with them. In addition, it is likely that implementing the methods for the first few projects will be challenging. With those caveats in mind, the path should be blazed, because once the methods are familiar and reliable, new research opportunities and efficiencies will inevitably arise. Investigator teams without the time, resources, or patience to implement these new methods can ordinarily rely on conventional meta-analysis methods,⁹⁹ which solve the privacy protection concerns but also have some important disadvantages by comparison.^{97, 98}