1. Introduction

Registries have the potential to produce data that are an important source of information regarding healthcare patterns, decision making, and delivery, as well as the subsequent association of these factors with patient outcomes. Registries, for example, can provide valuable insight into the safety and/or effectiveness of an intervention or the efficiency, timeliness, quality, and patient centeredness of a healthcare system. Registry data may also be linked to other data sources, such as administrative health insurance claims databases, electronic health records (EHRs), or biorepositories, to investigate hypotheses or questions that are secondary to the original reason for data collection.

The utility and applicability of registry data and linked datasets (collectively, ‘registry data’) rely heavily on an understanding of how the data were derived and why they were recorded, and then on the quality of the data analysis plan and its users’ ability to interpret the results. Analysis and interpretation of these data begin with a series of core questions:

• Study purpose: Is the intent descriptive or comparative and does it address natural history, effectiveness, safety or other characterization?
• Patient population: Who was studied and how did they come to be included in the registry?
• Data quality: How were the data collected and reviewed, and was any verification or validation performed?
• Data completeness: How were missing data handled for the main exposures and outcomes of interest and main confounders?
• Data analysis: What analyses were performed and how were differences in risk factors accounted for? Were any sensitivity analyses conducted to estimate the impact of bias on the observed results?

While registry data present many opportunities for meaningful analysis, there are inherent challenges to making appropriate inferences. Principal concerns include availability of the key variables of interest as well as data quality, since registries and secondary data sources vary in terms of their data quality assurance procedures and information on data quality or curation is not reported consistently. More importantly, nonrandomized studies of all designs are susceptible to systematic errors arising from mismeasurement of analytic variables,¹ unmeasured confounding,² and poor choice of reference group.^3,4 These factors must be considered when making inferences based on analyses of registry data.⁵

This chapter explains how analysis plans are constructed for registry data, how they differ depending on the purpose of the analysis, and how registry design and conduct and the characteristic(s) of any linked data can affect analysis and interpretation. The analytic techniques generally used for registry data are presented, addressing how conclusions may be drawn from the data and what caveats are appropriate. The chapter also describes how timelines for data analysis can be built in at registry inception and how to determine when the registry data are sufficient to begin analysis. Case Examples 23, 24, and 25 provide examples of registry analyses. Chapters 6 and 11 provide more information on the use of secondary data sources in registries.

2. Research Questions and Registry Purposes

Every registry study should start with a research question or focus. For example, disease registries commonly provide descriptive information, such as the typical clinical features of individuals with a disease, variations in phenotype, and the path to diagnosis and clinical progression of the disease over time (i.e., natural history) but they may also compare the effectiveness and safety of various treatments. These registries play a particularly important role in the study of many conditions, and especially for rare diseases.

In the case of studies where the aim is to examine the associations between specific exposures and outcomes, there is a vocal school of thinking that requires prespecification (and registration announcing such prespecification) of the study hypotheses or research questions as a requirement of assuring credible results.⁶ While aimed at avoiding publication bias, there are many strong counterarguments.^7–10 The key area of agreement centers around the importance of transparency and making study protocols including analytic plans available for review and publishing results with enough detail to allow replication for confirmation or refutation.

Regardless of whether the hypotheses are prespecified, most productive registry-based research begins with a clear statement of objectives. These objectives might be descriptive or may involve a comparison. Some examples of objectives include:

• Measure the incidence of a disease in a specific population,
• Characterize the patterns or costs of treatment for a disease in a specific population
• Measure the occurrence of outcomes among patients with a disease.
• Compare the incidence of a particular disease in two or more subgroups defined by common characteristics (e.g., etiologic research)
• Compare the cost or quality of care for a particular disease in two or more subgroups (e.g., health services research or disparities research)
• Compare the rate of outcomes among two or more subgroups of patients (often defined by different types or levels of treatment) with a particular disease (e.g., clinical research)

In all cases, the overarching objective is to obtain an accurate and valid estimate of the frequency of an outcome’s occurrence, or its relative frequency compared across groups.¹¹ An additional objective may be to generalize study results to a broader population. A valid estimate is one that is likely to be true. A precise estimate is one that has little variability. A generalizable estimate is one that provides information pertinent to the target population, the population for which the study’s information provides a basis for potential action, such as a public health or medical intervention. This is discussed further in the following section.

3. Patient Population

The purpose of registry-based research is to provide information about a specific patient population to which all study results are meant to apply. To determine how well the study results apply to the target population, five populations, each of which is a subset of the preceding population, need to be considered, along with how well each population represents the preceding population. These five subpopulations are shown in Figure 13-1.

Figure 13-1

Patient populations.

The target population is defined by the study’s purpose. To assess the appropriateness of the target population, one must ask the question, “Is this really the population that we need to know about?” For example, the target population for a registry of oral contraceptive users would include women of childbearing age who could become pregnant and are seeking to prevent pregnancy. Studies often miss important segments of the population in an effort to make the study population more homogeneous. For example, a study to assess a medical device used to treat patients for cardiac arrhythmias that defines only men as its target population would be less informative than it could be, because the device is designed for use in both men and women. Studies using linked datasets may miss segments of the population because of limitations in the secondary data source; for example, a study that linked registry data to claims data from a private payer would miss patients covered by other payers (e.g., Medicare) as well as the uninsured.

The accessible population is defined using inclusion criteria and exclusion criteria. The inclusion criteria define the population that will be used for the study and generally include geographic (e.g., hospitals or clinics in the New England region), demographic, disease-specific, and temporal (e.g., specification of the included dates of hospital or clinic admission), as well as other criteria. Conversely, the exclusion criteria seek to eliminate specific patients from study and may be driven by an effort to assure an adequate-sized population of interest for analysis. The same goals may be said of inclusion criteria, since it is difficult to separate inclusion from exclusion criteria (e.g., inclusion of adults aged 18 and older vs. exclusion of children younger than 18).

The accessible population may lose representativeness to the extent that convenience plays a part in its determination, because people who are easy to enroll in the registry or capture in secondary data sources may differ in some critical respects from the population at large. Similarly, to the extent that homogeneity plays a part in determining the accessible population, it is less likely to be representative of the entire population because certain population subgroups will be excluded.

Factors to be considered in assessing the accessible population’s representativeness of the target population include all the inclusion and exclusion criteria mentioned above. One method of evaluating representativeness is to describe the demographics and other key descriptors of the study population and to contrast its composition with patients with similar characteristics who are identified from another database, such as might be obtained from health insurers, health maintenance organizations, or the U.S. Surveillance Epidemiology and End Results (SEER) cancer registries.¹² For example, the Get With The Guidelines (GWTG)-Stroke registry was linked with Medicare claims data to examine the representativeness of the registry’s population of Medicare beneficiaries admitted for ischemic stroke.¹³

However, simple numerical/statistical representativeness is not the main issue. Representativeness should be evaluated in the context of the purpose of the study—that is, whether the study results can reasonably be generalized or extrapolated to other populations of interest outside of those included in the accessible population. For example, suppose that the purpose of the study is to assess the effectiveness of a drug in U.S. residents with diabetes. If the accessible population includes no children, then the study results may not apply to children, since children often metabolize drugs very differently from adults.

On the other hand, consider the possibility that the accessible population is generally drawn from a geographically isolated region, whereas the target population may be the entire country or the world. In that case, the accessible population is not geographically representative of the target population, but that circumstance would have little or no impact on the representativeness of the study findings to the target population if the action of the drug (or its delivery) does not vary geographically (which we would generally expect to be the case, unless pertinent racial/genetic or dietary factors were involved or if risk factors for the outcome differ geographically). Therefore, in this example, the lack of geographical representativeness would not affect interpretation of results.¹⁴ In common practice, representativeness is interpreted to draw accessible populations that represent typical patients and practitioners, rather than referring to representative samples in the statistical sense of the term.

The reason for using an intended population rather than the whole accessible population for the study is simply a matter of convenience and practicality. The issues to consider in assessing how well the intended population represents the accessible population are similar to those for assessing how well the accessible population represents the target population. The main difference is that the intended population may be specified by a sampling scheme, which often tries to strike a balance among representativeness, convenience, and budget. If the study is designed to estimate a rate of occurrence of in a specific population, then a random sample of the accessible population would be considered representative of the accessible population. It is important to note that for many, if not most, registry-based studies, a complete roster of the accessible population does not exist outside of single health system. More commonly, the intended population is compared with the accessible population in terms of pertinent variables.

To the extent that convenience or other design (e.g., stratified random sample) is used to choose the intended population, one must consider the extent to which the sampling of the accessible population may affect any interpretations from the study. For example, suppose that, for the sake of convenience, only patients who attend clinic on Mondays are included in the study. If patients who attend clinic on Mondays are similar in every relevant respect to other patients, that may not constitute a limitation. But if Monday patients are substantially different from patients who attend clinic on other days of the week (e.g., well-baby clinics are held on Mondays) and if those differences affect the outcome that is being studied (e.g., proportion of baby visits for “well babies”), then that sampling strategy would substantially alter the interpretations from the study and would be considered a meaningful limitation.

The extent to which the actual population is not representative of the intended population (or typical patients and/or practitioners) is generally a matter of real-world issues that prevent recruitment of a more comprehensive population. It is important to consider the likely underlying factors that caused those subjects not to be included in the analysis of study results and how that might affect the interpretations from the registry. For example, consider a study of a newly introduced medication, such as an anti-inflammatory drug that is thought to be as effective as other products and to have fewer side effects but that is more costly. Inclusion in the actual population may be influenced by prescribing practices governed by a health insurer. For example, if a new drug is approved for reimbursement only for patients who have “failed” treatment with other anti-inflammatory products, the resulting actual population will be systematically different from the target population of potential anti-inflammatory drug users. The actual population may be refractory to treatment or may have more comorbidities (e.g., gastrointestinal problems), and may be specifically selected for treatment beyond the intention of the study-specified inclusion criteria. In fact, registries of newly introduced drugs and devices may often include patients who are different from the ultimate target population of broad interest.

A related issue is that bias that could result from recruitment of early adopters,¹⁵ in which practitioners who are quick to use a novel healthcare intervention or therapy differ from those who use it only once it is well established. For example, a study of the use of a new surgical technique may initially enroll largely academic physicians and only much later enroll community-based surgeons. If the outcomes of the technique differ between the academic surgeons (early adopters) and community-based surgeons (later adopters), then the initial results of the study may not reflect the true effectiveness of the technique in widespread use. In fact, “operator experience” is an important factor in understanding the effectiveness of various surgical approaches or of devices that require surgical implantation. Patients selected for treatment with a novel therapy may also differ with regard to factors such as severity or duration of disease and prior treatment history, including treatment failures. For example, patients with more severe or late-stage disease who have failed other treatments might be more likely to use a newly approved product that has shown efficacy in treating their condition. Later on, patients with less severe disease may start using the product.

Finally, the analytic population includes all those patients who meet the criteria for analysis. In some cases, it becomes apparent that there are too few cases of a particular type, or too few patients with certain attributes, such that these subgroups do not contribute enough information for meaningful analysis. Analytic populations are also created to meet specific needs. For example, an investigator may request a dataset that will be used to analyze a subset of the registry population, such as those who had a specific treatment or condition.

Patients who are included in the analytic population for a given analysis may also be subject to selection or inclusion criteria (admissibility criteria), and these may affect interpretation of the resulting analyses. For example, if only patients who survive long enough to be admitted to hospital are included in a study, then immortal time bias may bias the results.¹⁶ Another example is when patients who remain enrolled in a registry and attend followup visits through 2 years after registry initiation are included in analysis of adherence to therapy. In this event, it is possible or likely that adherence among those who remain enrolled in the registry will be different from adherence among those who do not. Differential loss to followup, whereby patients who are lost may be more likely to experience adverse outcomes, such as mortality, than those who remain under observation, is a related issue that may lead to biased results. (See Chapter 3.)

Selection of a study population inevitably involves balancing accuracy and generalizability concerns, as well as cost and feasibility considerations. For example, restriction is one of the most effective strategies for control of confounding through study design.¹⁷ If one is concerned about confounding by sex, a simple and effective strategy to control that confounding is to restrict the study population to a single sex. However, such restriction reduces the study’s precision by decreasing the sample size, and may also reduce the generalizability of the results (only applicable to a segment of the target population). An alternative would be to include both sexes and to stratify the analysis by sex. While this approach would improve the generalizability of the results and allow for an evaluation of confounding, the precision of the estimated association would be reduced, and perhaps substantially reduced, if the estimate of effect in men was substantially different from the estimate of effect in women. In this circumstance, the study becomes effectively two studies.

4. Data Quality

In addition to a full understanding of study design and methodology, analysis of registry events and outcomes will benefit from an assessment of data quality. When registry data are linked to secondary data sources, this quality assessment must consider both the quality of the registry data as well as the original purpose, inherent limitations, likelihood of differential followup, and quality of any data linked to the registry data; the accuracy of matching the data to specific patients should also be considered. One must examine whether most if not all important covariates were collected, how analytic variables were defined in registry and secondary data sources, data completeness for key variables of interest, how missing data were handled, and data accuracy.

Linked datasets may offer the opportunity to validate some registry data. For example, pharmacy data can be used to confirm that prescriptions were actually filled and may provide more accurate identification of medication use than the data recorded in a registry. The frequency of pharmacy refills is often an indicator of adherence and may be more accurate than patient-reported adherence. Similarly, registries that are derived primarily from patient-reported data may use record linkage for clinical validation of events of special interest or to supplement patient-reported information with clinical or administrative data.

5. Data Analysis

Statistical methods commonly used for descriptive purposes include those that summarize information from continuous variables (e.g., mean, median) or from categorical variables (e.g., proportions, rates). Registries may describe a population using incidence (the proportion of the population that develops the condition over a specified time interval) and prevalence (the proportion of the population that has the condition at a specific point in time). Another summary estimate that is often used is an incidence rate. The incidence rate (also known as absolute risk) takes into account both the number of people in a population who develop the outcome of interest and the person-time at risk, or the length of time contributed by all people during the period when they were in the population and the events were counted.

For analytical studies, the association between a risk factor and outcome may be expressed as relative risk, odds ratio, or hazard ratio, depending on the nature of the data collected, the duration of the study, and the frequency of the outcome. The standard textbooks cited here have detailed discussions regarding epidemiologic and statistical methods commonly used for the various analyses supported by registries.^18,32–35

It is always important to consider the role of confounding. Although those planning a study try to collect as much data as possible to address known confounders, there is always the chance that unknown or unmeasured confounders will affect the interpretation of analyses derived from observational studies. It is important to consider the extent to which bias (systematic error stemming from factors that are related to both the decision to treat and the outcomes of interest [confounders]) could have distorted the results. For example, a bias known as confounding by indication³⁶ results from the fact that physicians do not prescribe medicine at random: the reason a patient is put on a particular regimen is often associated with their underlying disease severity and may, in turn, affect treatment outcome. To detect such a bias, the distribution of various prognostic factors at baseline is compared for patients who receive a treatment of interest and those who do not. A related concept is channeling bias, in which drugs with similar therapeutic indications are prescribed to groups of patients who may differ with regard to factors influencing prognosis.³⁷ Other types of bias include detection bias³⁸ (e.g., when comparison groups are assessed at different points in time or by different methods), selective loss to followup in which patients with the outcomes of most interest (e.g., sickest) may differentially drop out of one treatment group than another, and performance bias (e.g., systematic differences in care other than the intervention under study, such as a public health initiative promoting healthy lifestyles directed at patients who receive a particular class of treatment). In addition to such biases, analyses need to account for effect modification.³⁹ The presence of effect modification may also be identified after the data are collected.

Confounding may be evaluated using stratified analysis, multivariable analysis, sensitivity analyses, and simple or quantitative bias analysis or simply by graphical comparison of characteristics and events between groups.⁴⁰ The extensive information and large sample sizes available in some registries also support use of more advanced modeling techniques for addressing confounding by indication, such as the use of propensity scores to create matched comparison groups, or for stratification or inclusion in multivariable risk modeling.^41–44 New methods also include the high-dimensional propensity score (hd-PS) for adjustment using administrative data.⁴⁵ Examples are too numerous for a few selections to be fully representative, but registries in nearly every therapeutic area, including cancer,⁴⁶ cardiac devices,⁴⁷ organ transplantation,⁴⁸ and rare diseases,⁴⁹ have published the results of analyses incorporating approaches based on propensity scores. Nonetheless, application of these and other methods may not fully control for unmeasured confounding.⁵⁰

Groupings within a study population, such as patients seen by a single clinician or practice, residents of a neighborhood, or other “clusters,” may themselves impact or predict health outcomes of interest. Such groupings may be accounted for in analysis through use of analytic methods including analysis of variance (ANOVA), and hierarchical or multilevel modeling.^51–54

Heterogeneity of treatment effect is also an important consideration for comparative effectiveness research as the effect of a treatment may vary within subgroups of heterogeneous patients.⁵⁵ Stratification on the propensity score has been used to identify heterogeneity of treatment effect and may identify clinically meaningful differences between subgroups based on pre-treatment characteristics.

For economic analyses, the analytic approaches often encountered are cost-effectiveness analyses and cost-utility studies. To examine cost-effectiveness, costs are compared with clinical outcomes measured in units such as life expectancy or years of disease avoided.⁵⁶ Cost-utility analysis, a closely related technique, compares costs with outcomes adjusted for quality of life (utility) using measures known as quality-adjusted life years. Since most new interventions are more effective but also more expensive, another analytic approach examines the incremental cost-effectiveness ratio and contrasts that to the willingness to pay. (Willingness-to-pay analyses are generally conducted on a country-by-country basis, since various factors relating to national health insurance practices and cultural issues affect willingness to pay.) The use of registries for cost-effectiveness evaluations is a fairly recent development, and consequently, the methods are evolving rapidly. More information about economic analyses can be found in standard textbooks.^57–62

A number of biostatistics and epidemiology textbooks cover in depth the issues raised in this section and the appropriate analytic approaches for addressing them—for example, “time-to-event” or survival analyses⁶³ and issues of recurrent outcomes and repeated measures, with or without missing data,⁶⁴ in longitudinal cohort studies. Other texts address a range of regression and nonregression approaches to analysis of case-control and cohort study designs⁶⁵ that may be applied to registries. For further information on how to quantify bias, please see Lash, Fox, and Fink.²

5.3. Timing of Analyses During the Study

Unlike a typical clinical trial, registries, especially those that take several years to complete, may conduct intermediate analyses before all patients have been enrolled and/or all data collection has been completed. Such midcourse analyses may be undertaken for several reasons. First, many of these registries focus on serious safety outcomes. For such safety studies, it is important for all parties involved to actively monitor the frequency of such events at regular predefined intervals so that further risk assessment or risk management can be considered. The timing of such analyses may be influenced by regulatory requirements. Second, it may be of interest to examine treatment practices or health outcomes during the study to capture any emerging trends. Finally, it may also be important to provide intermediate or periodic analysis to document progress, often as a requirement for continued funding.

While it is useful to conduct such periodic analysis, careful planning should be given to the process and timing and whether the need for such analyses are programmatic (e.g., for annual progress reports) or for scientific purposes. For scientific purposes, among the first questions generally asked is whether a sufficient number of patients have been enrolled, sufficient followup time has elapsed to observe events of interest and/or whether a sufficient number of events have occurred. For example, some events, such as site reactions to injections, can be observed after a relatively short duration, compared with events like cancers, which may have a long induction or latency. If there are too few patients or insufficient time has elapsed, premature analyses may lead to the unreliable conclusions. However, it is inappropriate to delay analysis so long that an opportunity might be missed to observe emerging safety outcomes. Investigators should use sound clinical and epidemiological judgment when planning interim or periodic analyses and be sure that any resultant reports address both the strengths and limitations of such analyses.

5.3.1. Patient Censoring

At the time of a registry analysis, events may not have occurred for all patients. For these patients, the data are said to be censored, indicating that the observation period of the registry was stopped before all events occurred (e.g., mortality). In these situations, it is unclear when the event will occur, if at all. In addition, a registry may enroll patients until a set stop date, and patients entered into the registry earlier will have a greater probability of having an event than those entered more recently because of the longer followup. An important assumption, and one that needs to be assessed in a registry, is how patient prognosis varies with the time of entrance into the registry. This issue may be particularly problematic in registries that assess innovative (and changing) therapies. Patients and outcomes initially observed in the registry may differ from patients and outcomes observed later in the registry timeframe, either because of true differences in treatment options available at different points in time, or because of the shorter followup for people who entered later. Patients with censored data, however, contribute important information to the registry analysis. When possible, analyses should be planned so as to include all subjects, including those censored before the end of the followup period or the occurrence of an event. One method of analyzing censored data to estimate the conditional probability of the event occurring is to use the Kaplan-Meier method.⁷² In this method, for each time period, the probability is calculated that those who have not experienced an event before the beginning of the period will still not have experienced it by the end of the period. The probability of an event occurring at any given time is then calculated from the product of the conditional probabilities of each time interval.

5.3.2. Sensitivity Analyses

Sensitivity analysis refers to a procedure used to determine how robust the study result is to alterations of various parameters. If a small parameter alteration leads to a relatively large change in the results, the results are said to be sensitive to that parameter. Sensitivity and bias analyses may be used to determine how the final study results might change when taking into account those lost to followup. A hypothetical simple sensitivity analysis is presented in Table 13-1.

Table 13-1

Impact of loss to followup on incidence rates per 1,000 in a hypothetical study of 1,000 patients in a registry. Assuming that the incidence rate among patients lost to followup is X times the rate of incidence estimated in those who stayed in the registry: (more...)

Table 13-1 illustrates the extent of change in the incidence rate of a hypothetical outcome assuming varying degrees of loss to followup, and differences in incidence between those for whom there is information and those for whom there is no information due to loss to followup. In the first example, where 10 percent of the patients are lost to followup, the estimated incidence rate of 111/1,000 people is reasonably stable; it does not change too much when the (unknown) incidence in those lost to followup changes from 0.5 times the observed to 3 times the observed, with the corresponding incidence rate that would have been observed ranging from 106 to 156 per 1,000. On the other hand, when the loss to followup increases to 30 percent, the corresponding incidence rates that would have been observed range from 94 to 242. This procedure could be extended to a study that has more than one cohort of patients, with one being exposed and the other being nonexposed. In that case, the impact of loss to followup on the relative risk could be estimated by using sensitivity analysis.

6. Interpretation and Reporting of Registry Data

Proper interpretation of registry data is grounded in a strong understanding of the strengths and limitations of the registry methods including the analyses. Interpretation also should also be tempered, in part, by whether the analyses are attempting to confirm or refute other studies or if the registry is providing first reports. If the purpose of the registry is explicit, the actual population studied is reasonably representative of the target population, the data have been curated to enhance quality, and the analyses performed so as to reduce potential biases, then the interpretation of the registry data should allow a realistic picture of the quality of medical care, the natural history of the disease studied, or the safety, effectiveness, or value of a clinical evaluation. Each of these topics needs to be discussed in the interpretation of the registry data, and potential shortcomings should be explored.

In interpreting the findings, the precision of the estimated effect measure from the study should be discussed. Confidence intervals are important tools that provide a range of the effect measures consistent with the study findings. Statistical significance alone does not exclusively determine the clinical importance of the findings because some registries include large amounts of healthcare data, and very small effect measures can be statistically significant. Considerations should also be given to the clinical significance of the effect estimates and potential biases.⁹⁸

Assumptions or biases that could have influenced the outcomes of the analyses should be highlighted and separated from those that do not affect the interpretation of the registry results. Documentation of how data were collected and coded, data completeness and how missing data were addressed when reporting registry findings is important to provide transparency and to allow readers to accurately interpret registry findings. To this end, three useful guidelines are the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement,⁹⁹ the Patient-Centered Outcomes Research Institute (PCORI) Methodology Report,¹⁰⁰ and the GRACE checklist for observational studies of comparative effectiveness.¹⁰¹

7. Summary

In summary, a meaningful analysis requires careful consideration of study design features and the nature of the data collected. Most typical epidemiological study analytical methods can be applied, and there is no one-size-fits-all approach for registry-based research. Efforts should be made to carefully evaluate the presence of biases and to control for identified potential biases during data analysis. This requires close collaboration among clinicians, epidemiologists, statisticians, study coordinators, and others involved in the design, conduct, and interpretation of the study.

References for Chapter 13

1.

Rothman K, Greenland S, Lash TL. Validity in Epidemiologic Studies. In: Rothman K, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. p. 128–47.

2.

Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data: Springer; 2009.

3.

Rothman K, Greenland S, Poole C, et al. Causation and Causal Inference. In: Rothman K, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. p. 5–31.

4.

Rothman K, Greenland S, Lash TL. Case-Control Studies. In: Rothman K, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. p. 111–27.

5.

Sedrakyan A, Marinac-Dabic D, Normand SL, et al. A framework for evidence evaluation and methodological issues in implantable device studies. Med Care. 2010;48:(6 Suppl):S121–8. PMID: 20421824. DOI: 10.1097/MLR.0b013e3181d991c4. [PubMed: 20421824] [CrossRef]

6.

Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 2017;26(9):1033–9. PMID: 28913966. DOI: 10.1002/pds.4297. [PMC free article: PMC5639372] [PubMed: 28913966] [CrossRef]

7.

Editors. The registration of observational studies--when metaphors go bad. Epidemiology. 2010;21(5):607–9. PMID: 20657291. DOI: 10.1097/EDE.0b013e3181eafbcf. [PubMed: 20657291] [CrossRef]

8.

Poole C. A vision of accessible epidemiology. Epidemiology. 2010;21(5):616–8. PMID: 20657293. DOI: 10.1097/EDE.0b013e3181e9be3f. [PubMed: 20657293] [CrossRef]

9.

Lash TL. Preregistration of study protocols is unlikely to improve the yield from our science, but other strategies might. Epidemiology. 2010;21(5):612–3. PMID: 20657295. DOI: 10.1097/EDE.0b013e3181e9bba6. [PubMed: 20657295] [CrossRef]

10.

Lash TL, Collin LJ, Van Dyke ME. The Replication Crisis in Epidemiology: Snowball, Snow Job, or Winter Solstice? Curr Epidemiol Rep. 2018;5(2):175–83. [PMC free article: PMC8075285] [PubMed: 33907664]

11.

Greenland S, Rothman K. Measures of Occurrence. In: Rothman K, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. p. 32–50.

12.

National Cancer Institute. Surveillance Epidemiology and End Results. https://seer​.cancer.gov/. Accessed June 10, 2019.

13.

Reeves MJ, Fonarow GC, Smith EE, et al. Representativeness of the Get With The Guidelines-Stroke Registry: comparison of patient and hospital characteristics among Medicare beneficiaries hospitalized with ischemic stroke. Stroke. 2012;43(1):44–9. PMID: 21980197. DOI: 10.1161/STROKEAHA.111.626978. [PubMed: 21980197] [CrossRef]

14.

Rothman KJ, Gallacher JE, Hatch EE. Why representativeness should be avoided. Int J Epidemiol. 2013;42(4):1012–4. PMID: 24062287. DOI: 10.1093/ije/dys223. [PMC free article: PMC3888189] [PubMed: 24062287] [CrossRef]

15.

Schneeweiss S, Gagne JJ, Glynn RJ, et al. Assessing the comparative effectiveness of newly marketed medications: methodological challenges and implications for drug development. Clin Pharmacol Ther. 2011;90(6):777–90. PMID: 22048230. DOI: 10.1038/clpt.2011.235. [PubMed: 22048230] [CrossRef]

16.

Suissa S. Effectiveness of inhaled corticosteroids in chronic obstructive pulmonary disease: immortal time bias in observational studies. Am J Respir Crit Care Med. 2003;168(1):49–53. PMID: 12663327. DOI: 10.1164/rccm.200210-1231OC. [PubMed: 12663327] [CrossRef]

17.

Rothman K, Greenland S, Lash TL. Design strategies to improve study accuracy. In: Rothman K, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. p. 162–82.

18.

Rothman K, Greenland S. Modern Epidemiology. 2nd ed. Philadelphia: Lippincott Williams & Wilkins; 1998.

19.

Dreyer NA. Advancing a Framework for Regulatory Use of Real-World Evidence: When Real Is Reliable. Ther Innov Regul Sci. 2018;52(3):362–8. PMID: 29714575. DOI: 10.1177/2168479018763591. [PMC free article: PMC5944086] [PubMed: 29714575] [CrossRef]

20.

U.S. Food and Drug Administration. Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research. Standards for Data Management and Analytic Processes in the Office of Surveillance and Epidemiology (OSE). March 3, 2008. hhttps://www​.fda.gov/media/72761/download. Accessed June 10, 2019.

21.

Greenland S, Lash TL. Bias Analysis. In: Rothman K, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2008. p. 345–80.

22.

Sorensen HT, Riis AH, Lash TL, et al. Statin use and risk of amyotrophic lateral sclerosis and other motor neuron disorders. Circ Cardiovasc Qual Outcomes. 2010;3(4):413–7. PMID: 20530788. DOI: 10.1161/CIRCOUTCOMES.110.936278 [PubMed: 20530788] [CrossRef]

23.

Anderson IB, Sorensen TI, Prener A. Increase in incidence of disease due to diagnostic drift: primary liver cancer in Denmark, 1943–85. BMJ. 1991;302(6774):437–40. PMID: 2004170. DOI: 10.1136/bmj.302.6774.437. [PMC free article: PMC1669338] [PubMed: 2004170] [CrossRef]

24.

Lash TL, Johansen MB, Christensen S, et al. Hospitalization rates and survival associated with COPD: a nationwide Danish cohort study. Lung. 2011;189(1):27–35. PMID: 21170722. DOI: 10.1007/s00408-010-9274-z. [PubMed: 21170722] [CrossRef]

25.

Rodriguez LA, Perez-Gutthann S, Jick SS. The UK General Practice Research Database. In: Strom BL, editor. Pharmacopepidemiology. 3rd ed. Chichester, UK: John Wiley & Sons, LTD; 2000. p. 375–85.

26.

Greenland S, Schwartzbaum JA, Finkle WD. Problems due to small samples and sparse data in conditional logistic regression analysis. Am J Epidemiol. 2000;151(5):531–9. PMID: 10707923. DOI: 10.1093/oxfordjournals.aje.a010240. [PubMed: 10707923] [CrossRef]

27.

Sturmer T, Jonsson Funk M, Poole C, et al. Nonexperimental comparative effectiveness research using linked healthcare databases. Epidemiology. 2011;22(3):298–301. PMID: 21464649. DOI: 10.1097/EDE.0b013e318212640c. [PMC free article: PMC4012640] [PubMed: 21464649] [CrossRef]

28.

Greenland S. Randomization, statistics, and causal inference. Epidemiology. 1990;1(6):421–9. PMID: 2090279. [PubMed: 2090279]

29.

Lash TL, Schmidt M, Jensen AO, et al. Methods to apply probabilistic bias analysis to summary estimates of association. Pharmacoepidemiol Drug Saf. 2010;19(6):638–44. PMID: 20535760. DOI: 10.1002/pds.1938. [PubMed: 20535760] [CrossRef]

30.

Fink AK, Lash TL. A null association between smoking during pregnancy and breast cancer using Massachusetts registry data (United States). Cancer Causes Control. 2003;14(5):497–503. PMID: 12946045. [PubMed: 12946045]

31.

Lash TL, Fox MP, Thwin SS, et al. Using probabilistic corrections to account for abstractor agreement in medical record reviews. Am J Epidemiol. 2007;165(12):1454–61. PMID: 17406006. DOI: 10.1093/aje/kwm034. [PubMed: 17406006] [CrossRef]

32.

Kleinbaum DG, Kupper LL, Miller KE, et al. Applied regression analysis and other multivariable methods. Belmont, CA: Duxbury Press; 1998.

33.

Hennekens CH, Buring JE, Mayrent SL. Epidemiology in medicine. 1st ed. Boston: Little, Brown and Company; 1987.

34.

Aschengrau A, Seage G. Essentials of epidemiology in public health. 2003.

35.

Rosner B. Fundamentals of biostatistics. 5th ed. Boston: Duxbury Press; 2000.

36.

Salas M, Hofman A, Stricker BH. Confounding by indication: an example of variation in the use of epidemiologic terminology. Am J Epidemiol. 1999;149(11):981–3. PMID: 10355372. DOI: 10.1093/oxfordjournals.aje.a009758. [PubMed: 10355372] [CrossRef]

37.

Petri H, Urquhart J. Channeling bias in the interpretation of drug effects. Stat Med. 1991;10(4):577–81. PMID: 2057656. [PubMed: 2057656]

38.

Higgins J, Green S. The Cochrane Collaboration. The Cochrane handbook for systematic reviews of interventions. 2006. http://www​.cochrane.org​/sites/default/files​/uploads/Handbook4.2.6Sep2006.pdf. Accessed August 15, 2012.

39.

Swihart BJ, Caffo B, James BD, et al. Lasagna plots: a saucy alternative to spaghetti plots. Epidemiology. 2010;21(5):621–5. PMID: 20699681. DOI: 10.1097/EDE.0b013e3181e5b06a. [PMC free article: PMC2937254] [PubMed: 20699681] [CrossRef]

40.

Hernan MA, Hernandez-Diaz S, Werler MM, et al. Causal knowledge as a prerequisite for confounding evaluation: an application to birth defects epidemiology. Am J Epidemiol. 2002;155(2):176–84. PMID: 11790682. DOI: 10.1093/aje/155.2.176. [PubMed: 11790682] [CrossRef]

41.

Mangano DT, Tudor IC, Dietzel C, et al. The risk associated with aprotinin in cardiac surgery. N Engl J Med. 2006;354(4):353–65. PMID: 16436767. DOI: 10.1056/NEJMoa051379. [PubMed: 16436767] [CrossRef]

42.

Cepeda MS, Boston R, Farrar JT, et al. Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders. Am J Epidemiol. 2003;158(3):280–7. PMID: 12882951. DOI: 10.1093/aje/kwg115. [PubMed: 12882951] [CrossRef]

43.

Sturmer T, Joshi M, Glynn RJ, et al. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol. 2006;59(5):437–47. PMID: 16632131. DOI: 10.1016/j.jclinepi.2005.07.004. [PMC free article: PMC1448214] [PubMed: 16632131] [CrossRef]

44.

Glynn RJ, Schneeweiss S, Sturmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. 2006;98(3):253–9. PMID: 16611199. DOI: 10.1111/j.1742-7843.2006.pto_293.x. [PMC free article: PMC1790968] [PubMed: 16611199] [CrossRef]

45.

Schneeweiss S, Rassen JA, Glynn RJ, et al. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology. 2009;20(4):512–22. PMID: 19487948. DOI: 10.1097/EDE.0b013e3181a663cc. [PMC free article: PMC3077219] [PubMed: 19487948] [CrossRef]

46.

Reeve BB, Potosky AL, Smith AW, et al. Impact of cancer on health-related quality of life of older Americans. J Natl Cancer Inst. 2009;101(12):860–8. PMID: 19509357. DOI: 10.1093/jnci/djp123. [PMC free article: PMC2720781] [PubMed: 19509357] [CrossRef]

47.

Brodie BR, Stuckey T, Downey W, et al. Outcomes with drug-eluting stents versus bare metal stents in acute ST-elevation myocardial infarction: results from the Strategic Transcatheter Evaluation of New Therapies (STENT) Group. Catheter Cardiovasc Interv. 2008;72(7):893–900. PMID: 19016465. DOI: 10.1002/ccd.21767. [PubMed: 19016465] [CrossRef]

48.

Shuhaiber JH, Kim JB, Hur K, et al. Survival of primary and repeat lung transplantation in the United States. Ann Thorac Surg. 2009;87(1):261–6. PMID: 19101309. DOI: 10.1016/j.athoracsur.2008.10.031. [PubMed: 19101309] [CrossRef]

49.

Grabowski GA, Kacena K, Cole JA, et al. Dose-response relationships for enzyme replacement therapy with imiglucerase/alglucerase in patients with Gaucher disease type 1. Genet Med. 2009;11(2):92–100. PMID: 19265748. DOI: 10.1097/GIM.0b013e31818e2c19. [PMC free article: PMC3793250] [PubMed: 19265748] [CrossRef]

50.

Hernan MA, Robins JM. Instruments for causal inference: an epidemiologist’s dream? Epidemiology. 2006;17(4):360–72. PMID: 16755261. DOI: 10.1097/01.ede.0000222409.00878.37. [PubMed: 16755261] [CrossRef]

51.

Merlo J, Chaix B, Yang M, et al. A brief conceptual tutorial of multilevel analysis in social epidemiology: linking the statistical concept of clustering to the idea of contextual phenomenon. J Epidemiol Community Health. 2005;59(6):443–9. PMID: 15911637. DOI: 10.1136/jech.2004.023473. [PMC free article: PMC1757045] [PubMed: 15911637] [CrossRef]

52.

Holden JE, Kelley K, Agarwal R. Analyzing change: a primer on multilevel models with applications to nephrology. Am J Nephrol. 2008;28(5):792–801. PMID: 18477842. DOI: 10.1159/000131102. [PMC free article: PMC2613435] [PubMed: 18477842] [CrossRef]

53.

Diez-Roux AV. Multilevel analysis in public health research. Annu Rev Public Health. 2000;21:171–92. PMID: 10884951. DOI: 10.1146/annurev.publhealth.21.1.171. [PubMed: 10884951] [CrossRef]

54.

Leyland AH, Goldstein H. Multilevel modeling of health statistics. Chichester, UK: John Wiley & Sons, LTD; 2001.

55.

Varadhan R, Segal JB, Boyd CM, et al. A framework for the analysis of heterogeneity of treatment effect in patient-centered outcomes research. J Clin Epidemiol. 2013;66(8):818–25. PMID: 23651763. DOI: 10.1016/j.jclinepi.2013.02.009. [PMC free article: PMC4450361] [PubMed: 23651763] [CrossRef]

56.

Palmer AJ. Health economics--what the nephrologist should know. Nephrol Dial Transplant. 2005;20(6):1038–41. PMID: 15840678. DOI: 10.1093/ndt/gfh824. [PubMed: 15840678] [CrossRef]

57.

Neumann PJ. Opportunities and barriers. Using cost-effectiveness analysis to improve health care. New York, NY: Oxford University Press; 2004.

58.

Edejer TT-T, Baltussen R, Adam T, et al. Making choices in health: WHO guide to cost-effectiveness analysis. Geneva: World Health Organization; 2004.

59.

Drummond M, Stoddart G, Torrance G. Methods for the economic evaluation of health care programmes. 3rd ed. New York: Oxford University Press; 2005.

60.

Muennig P. Designing and conducting cost-effectiveness analyses in medicine and health care. New York: John Wiley & Sons, LTD; 2002.

61.

Haddix AC, Teutsch SM, Corso PS. Prevention effectiveness: a guide to decision analysis and economic evaluation. New York: Oxford University Press; 2003.

62.

Gold MR, Siegel JE, Russell LB, et al. Cost-effectiveness in health and medicine: the Report of the Panel on Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996.

63.

Kleinbaum DG, Klein M. Survival analysis: a self-learning text. 2nd ed. New York: Springer; 2005.

64.

Twisk JWR. Applied longitudinal data analysis for epidemiology – a practical guide. Cambridge, UK: Cambridge University Press; 2003.

65.

Newman SC. Biostatistical methods in epidemiology. New York: John Wiley & Sons, LTD; 2001.

66.

Ray WA. Evaluating medication effects outside of clinical trials: new-user designs. Am J Epidemiol. 2003;158(9):915–20. PMID: 14585769. DOI: 10.1093/aje/kwg231. [PubMed: 14585769] [CrossRef]

67.

Suissa S. Immortal time bias in observational studies of drug effects. Pharmacoepidemiol Drug Saf. 2007;16(3):241–9. PMID: 17252614. DOI: 10.1002/pds.1357. [PubMed: 17252614] [CrossRef]

68.

Haynes RB, Sackett DL, Guyatt GH, et al. Clinical epidemiology. 3rd ed. New York: Lippincott Williams & Wilkens; 2005.

69.

Andersen PK, Geskus RB, de Witte T, et al. Competing risks in epidemiology: possibilities and pitfalls. Int J Epidemiol. 2012;41(3):861–70. PMID: 22253319. DOI: 10.1093/ije/dyr213. [PMC free article: PMC3396320] [PubMed: 22253319] [CrossRef]

70.

Greenland S, Pearl J, Robins JM. Causal diagrams for epidemiologic research. Epidemiology. 1999;10(1):37–48. PMID: 9888278. [PubMed: 9888278]

71.

Hernan MA, Hernandez-Diaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15(5):615–25. PMID: 15308962. [PubMed: 15308962]

72.

Bland JM, Altman DG. Survival probabilities (the Kaplan-Meier method). BMJ. 1998;317(7172):1572. PMID: 9836663. DOI: 10.1136/bmj.317.7172.1572. [PMC free article: PMC1114388] [PubMed: 9836663] [CrossRef]

73.

Mendelsohn AB, Dreyer NA, Mattox PW, et al. Characterization of Missing Data in Clinical Registry Studies. Therapeutic Innovation & Regulatory Science. 2015;49(1):146–54. PMID: 30222467. DOI: 10.1177/2168479014532259. [PubMed: 30222467] [CrossRef]

74.

Applebaum KM, Malloy EJ, Eisen EA. Left truncation, susceptibility, and bias in occupational cohort studies. Epidemiology. 2011;22(4):599–606. PMID: 21543985. DOI: 10.1097/EDE.0b013e31821d0879. [PMC free article: PMC4153398] [PubMed: 21543985] [CrossRef]

75.

Howards PP, Hertz-Picciotto I, Poole C. Conditions for bias from differential left truncation. Am J Epidemiol. 2007;165(4):444–52. PMID: 17150983. DOI: 10.1093/aje/kwk027. [PubMed: 17150983] [CrossRef]

76.

Cain KC, Harlow SD, Little RJ, et al. Bias due to left truncation and left censoring in longitudinal studies of developmental and disease processes. Am J Epidemiol. 2011;173(9):1078–84. PMID: 21422059. DOI: 10.1093/aje/kwq481. [PMC free article: PMC3121224] [PubMed: 21422059] [CrossRef]

77.

Daniel RM, Kenward MG, Cousens SN, et al. Using causal diagrams to guide analysis in missing data problems. Stat Methods Med Res. 2012;21(3):243–56. PMID: 21389091. DOI: 10.1177/0962280210394469. [PubMed: 21389091] [CrossRef]

78.

Hernandez-Diaz S, Chambers C, Ephross S, et al. “Pregnancy Registries.” In: Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014. http://www​.effectivehealthcare.ahrq.gov. [PubMed: 24945055]

79.

Lau B, Cole SR, Gange SJ. Competing risk regression models for epidemiologic data. Am J Epidemiol. 2009;170(2):244–56. PMID: 19494242. DOI: 10.1093/aje/kwp107. [PMC free article: PMC2732996] [PubMed: 19494242] [CrossRef]

80.

Riley GF. Administrative and claims records as sources of health care cost data. Med Care. 2009;47(7 Suppl 1):S51–5. PMID: 19536019. DOI: 10.1097/MLR.0b013e31819c95aa. [PubMed: 19536019] [CrossRef]

81.

Suissa S. Immeasurable time bias in observational studies of drug effects on mortality. Am J Epidemiol. 2008;168(3):329–35. PMID: 18515793. DOI: 10.1093/aje/kwn135. [PubMed: 18515793] [CrossRef]

82.

Bradbury BD, Wang O, Critchlow CW, et al. Exploring relative mortality and epoetin alfa dose among hemodialysis patients. Am J Kidney Dis. 2008;51(1):62–70. PMID: 18155534. DOI: 10.1053/j.ajkd.2007.09.015. [PubMed: 18155534] [CrossRef]

83.

Little RJA, Rubin DB. Statistical analysis with missing data. New York: John Wiley & Sons; 1987.

84.

Rubin D. Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons, LTD; 1987.

85.

A Guide to Planning for Missing Data. Clinical Trials with Missing Data. p. 71–129.

86.

Allison PD. Handling Missing Data by Maximum Likelihood. Paper 312-2012, SAS Global Forum 2012. http://www​.statisticalhorizons​.com/wp-content​/uploads/MissingDataByML.pdf. Accessed June 10, 2019.

87.

Loughin TM. SAS® for Mixed Models, 2nd edition Edited by Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., and Schabenberger, O. Biometrics. 2006;62(4):1273–4. DOI: 10.1111/j.1541-0420.2006.00596_6.x. [CrossRef]

88.

Greenland S, Finkle WD. A critical look at methods for handling missing covariates in epidemiologic regression analyses. Am J Epidemiol. 1995;142(12):1255–64. PMID: 7503045. DOI: 10.1093/oxfordjournals.aje.a117592. [PubMed: 7503045] [CrossRef]

89.

Hernán MA, Robins JM (2019). Causal Inference. Boca Raton: Chapman & Hall/CRC, forthcoming.

90.

Westreich D. Berkson’s bias, selection bias, and missing data. Epidemiology. 2012;23(1):159–64. PMID: 22081062. DOI: 10.1097/EDE.0b013e31823b6296. [PMC free article: PMC3237868] [PubMed: 22081062] [CrossRef]

91.

Motwani M, Dey D, Berman DS, et al. Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis. European heart journal. 2017;38(7):500–7. PMID: 27252451. DOI: 10.1093/eurheartj/ehw188. [PMC free article: PMC5897836] [PubMed: 27252451] [CrossRef]

92.

Linden A, Yarnold PR. Combining machine learning and matching techniques to improve causal inference in program evaluation. J Eval Clin Pract. 2016;22(6):864–70. PMID: 27353301. DOI: 10.1111/jep.12592. [PubMed: 27353301] [CrossRef]

93.

Oermann EK, Rubinsteyn A, Ding D, et al. Using a Machine Learning Approach to Predict Outcomes after Radiosurgery for Cerebral Arteriovenous Malformations. Sci Rep. 2016;6:21161. PMID: 26856372. DOI: 10.1038/srep21161. [PMC free article: PMC4746661] [PubMed: 26856372] [CrossRef]

94.

Lotsch J, Ultsch A. Machine learning in pain research. Pain. 2018;159(4):623–30. PMID: 29194126. DOI: 10.1097/j.pain.0000000000001118. [PMC free article: PMC5895117] [PubMed: 29194126] [CrossRef]

95.

Al-Haddad MA, Friedlin J, Kesterson J, et al. Natural language processing for the development of a clinical registry: a validation study in intraductal papillary mucinous neoplasms. HPB (Oxford). 2010;12(10):688–95. PMID: 21083794. DOI: 10.1111/j.1477-2574.2010.00235.x. [PMC free article: PMC3003479] [PubMed: 21083794] [CrossRef]

96.

Glowacki J, Reichoff M. Effective Model Validation using Machine Learning. Milliman White Paper. May 2017. http://us​.milliman.com​/uploadedFiles/insight​/2017/effective-model-validation-machine-learning.pdf. Accessed June 10, 2019.

97.

Jones BE, South BR, Shao Y, et al. Development and Validation of a Natural Language Processing Tool to Identify Patients Treated for Pneumonia across VA Emergency Departments. Appl Clin Inform. 2018;9(1):122–8. PMID: 29466818. DOI: 10.1055/s-0038-1626725. [PMC free article: PMC5821510] [PubMed: 29466818] [CrossRef]

98.

U.S. Food and Drug Administration. Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data. May 2013. https://www​.fda.gov/files​/drugs/published​/Best-Practices-for-Conducting-and-Reporting-Pharmacoepidemiologic-Safety-Studies-Using-Electronic-Healthcare-Data-Sets.pdf, June 10, 2019.

99.

von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9. PMID: 25046131. DOI: 10.1016/j.ijsu.2014.07.013. [PubMed: 25046131] [CrossRef]

100.

Patient-Centered Outcomes Research Institute. PCORI Methodology Standards. April 2018. https://www​.pcori.org​/sites/default/files​/PCORI-Methodology-Standards.pdf. Accessed June 4, 2019.

101.

Dreyer NA, Bryant A, Velentgas P. The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. J Manag Care Spec Pharm. 2016;22(10):1107–13. PMID: 27668559. DOI: 10.18553/jmcp.2016.22.10.1107. [PMC free article: PMC10398313] [PubMed: 27668559] [CrossRef]

Case Examples for Chapter 13