1. Introduction

This chapter is intended as a high-level practical guide to the application of epidemiologic methods that are particularly useful in the design of registries that evaluate patient outcomes. Since it is not intended to replace a basic textbook on epidemiologic design, readers are encouraged to seek more information from textbooks and scientific articles. Table 3–1 summarizes the key considerations for study design that are discussed in this chapter. Throughout the design process, registry planners may want to discuss options and decisions with the registry stakeholders and relevant experts to ensure that sound decisions are made. The choice of groups to be consulted during the design phase generally depends on the nature of the registry, the registry funding source and funding mechanism, and the intended audience for registry reporting. More detailed discussions of registry design for specific types of registries are provided in Chapters 19, 20, 21, 22, and 23.

Table 3–1

Considerations for study design.

2. Research Questions Appropriate for Registries

The questions typically addressed in registries range from purely descriptive questions aimed at understanding the characteristics of people who develop the disease and how the disease generally progresses, to highly focused questions intended to support decisionmaking. Registries focused on determining clinical effectiveness or cost-effectiveness or assessing safety or harm are generally hypothesis driven and concentrate on evaluating the effects of specific treatments on patient outcomes. Research questions should address the registry's purposes, as broadly described in Table 3–2.

Table 3–2

Overview of registry purposes.

Observational studies derived from registries (or “registry-based studies”) are an important part of the research armamentarium alongside interventional studies, such as randomized controlled trials (RCTs) or pragmatic trials, and retrospective studies, such as studies derived exclusively from administrative claims data. Each of these study designs has strengths and limitations, and the selection of a study design should be guided by the research questions of interest. (See Chapter 2, Section 2.2, for a discussion of the factors that influence the study design decision.) In some cases, multiple studies with different designs or a hybrid study that combines study designs will be necessary to address a research question. In fact, this more comprehensive approach to evidence development is likely to become more common as researchers strive to address multiple questions for multiple stakeholders most efficiently. Observational studies and interventional studies are more complementary than competitive, precisely because some research questions are better answered by one method than the other. Interventional studies are considered by many to provide the highest grade evidence for evaluating whether a drug has the ability to bring about an intended effect in optimal or “ideal world” situations, a concept also known as “efficacy.”¹ Observational designs, on the other hand, are particularly well suited for studying broader populations, understanding actual results (e.g., some safety outcomes) in real-world practice (see Case Example 2), and for obtaining more representative quality-of-life information. This is particularly true when the factors surrounding the decision to treat are an important aspect of understanding treatment effectiveness.²

In many situations, nonrandomized comparisons either are sufficient to address the research question or, in some cases, may be necessary because of the following issues with randomizing patients to a specific treatment:

• Equipoise: Can providers ethically introduce randomization between treatments when the treatments may not be clinically equivalent?
• Ethics: If reasonable suspicion about the safety of a product has become known, would it be ethical to conduct a trial that deliberately exposes patients to potential harm? For example, can pregnant women be ethically exposed to drugs that may be teratogenic? (See Chapter 21 and Case Examples 49, 50, 51, and 52.)
• Practicality: Will patients enroll in a study where they might not receive the treatment, or might not receive what is likely to be the best treatment? How can compliance and adherence to a treatment be studied, if not by observing what people do in real-world situations?

Registries are particularly suitable for some types of research questions, such as:

• Natural history studies where the goal is to observe clinical practice and patient experience but not to introduce any intervention.
• Measures of clinical effectiveness, especially as related to compliance, where the purpose is to learn about what patients and practitioners actually do and how their actions affect real-world outcomes. This is especially important for treatments that have poor compliance.
• Studies of effectiveness and safety for which clinician training and technique are part of the study of the treatment (e.g., a procedure such as placement of carotid stent).
• Studies of heterogeneous patient populations, since unlike randomized trials, registries generally have much broader inclusion criteria and fewer exclusion criteria. These characteristics lead to studies with greater generalizability (external validity) and may allow for assessment of subgroup differences in treatment effects.
• Followup for delayed or long-term benefits or harm, since registries can extend over much longer periods than most clinical trials (because of their generally lower costs to run and lesser burden on participants).
• Surveillance for rare events or of rare diseases.
• Studies for treatments in which randomization is unethical, such as intentional exposure to potential harm (as in safety studies of marketed products that are suspected of being harmful).
• Studies for treatments in which randomization is not necessary, such as when certain therapies are only available in certain places owing to high cost or other restrictions (e.g., proton beam therapy).
• Studies for which blinding is challenging or unethical (e.g., studies of surgical interventions, acupuncture).
• Studies of rapidly changing technology.
• Studies of conditions with complex treatment patterns and treatment combinations.
• Studies of health care access and barriers to care.
• Evaluations of actual standard medical practice. (See Case Example 58.)

Registry studies may also include embedded substudies as part of their overall design. These substudies can themselves have various designs (e.g., highly detailed prospective data collection on a subset of registry participants, or a case-control study focused on either incident or prevalent cases identified within the registry). (See Case Examples 3 and 47.) Registries can also be used as sampling frames for RCTs.

3. Translating Clinical Questions Into Measurable Exposures and Outcomes

The specific clinical questions of interest in a registry will guide the definitions of study subjects, exposure, and outcome measures, as well as the study design, data collection, and analysis. In the context of registries, the term “exposure” is used broadly to include treatments and procedures, health care services, diseases, and conditions.

The clinical questions of interest can be defined by reviewing published clinical information, soliciting experts' opinions, and evaluating the expressed needs of the patients, health care providers, and payers. Examples of research questions, key outcome and exposure variables, and sources of data are shown in Table 3–3. As these examples show, the outcomes (generally beneficial or deleterious outcomes) are the main endpoints of interest posed in the research question. These typically represent measures of health or onset of illness or adverse events, but also commonly include quality of life measures, and measures of health care utilization and costs.

Table 3–3

Examples of research questions and key exposures and outcomes.

Relevant exposures also derive from the main research question and relate to why a patient might experience benefit or harm. Evaluation of an exposure includes collection of information that affects or augments the main exposure, such as dose, duration of exposure, route of exposure, or adherence. Other variables of interest include independent risk factors for the outcomes of interest (e.g., comorbidities, age), as well as variables known as potential confounding variables, that are related to both the exposure and the outcome and are necessary for conducting valid statistical analyses. Confounding can result in inaccurate estimates of association between the study exposure and outcome through mixing of effects. For example, in a study of asthma medications, prior history of treatment resistance should be collected or else results may be biased. The bias could occur because treatment resistance may relate both to the likelihood of receiving the new drug (meaning that doctors will be more likely to try a new drug in patients who have failed other therapies) and the likelihood of having a poorer outcome (e.g., hospitalization). Refer to Chapter 4 for a discussion of selecting data elements and Chapter 5 for a discussion of selecting patient-reported outcomes.

4. Finding the Necessary Data

The identification of key outcome and exposure variables and patients will drive the strategy for data collection, including the choice of data sources. A key challenge to registries is that it is generally not possible to collect all desired data. As discussed in Chapter 4, data collection should be both parsimonious and broadly applicable. For example, while experimental imaging studies may provide interesting data, if the imaging technology is not widely available, the data will not be available for enough patients to be useful for analysis. Moreover, the registry findings will not be generalizable if only sophisticated centers that have such technology participate. Instead, registries should focus on collecting relevant data with relatively modest burden on patients and clinicians. Registry data can be obtained from patients, clinicians, medical records, and linkage with other sources (in particular, extant databases), depending on the available budget. (See Chapters 6, 15, and 16.)

Examples of patient-reported data include health-related quality of life; utilities (i.e., patient preferences); symptoms; use of over-the-counter (OTC), complementary, and alternative medication; behavioral data (e.g., smoking and alcohol use); family history; and biological specimens. These data may rely on the subjective interpretation and reporting of the patient (e.g., health-related quality of life, utilities, symptoms such as pain or fatigue); may be difficult to otherwise track (e.g., use of complementary and alternative medication, smoking, and alcohol use); or may be unique to the patient (e.g., biological specimens). Health care resource utilization is another important construct that reflects both cost of care (burden of illness) and health-related quality of life. For example, more frequent office visits, procedures, or hospitalizations may result in reduced health-related quality of life for the patient. The primary advantage of this form of data collection is that it provides direct information from the entity that is ultimately of the most interest—the patient. The primary disadvantages are that the patient is not necessarily a trained observer and that various forms of bias, such as recall bias, may influence subjective information. For example, people may selectively recall certain exposures because they believe they have a disease that was caused by that exposure, or their recall may be influenced by recent news stories claiming cause-and-effect relationships. (See Case Example 4.)

Examples of clinician data include clinical impressions, clinical diagnoses, clinical signs, differential diagnoses, laboratory results, and staging. The primary advantage of clinician data is that clinicians are trained observers. Even so, the primary disadvantages are that clinicians are not necessarily accurate reporters of patient perceptions, and their responses may also be subject to recall bias. Moreover, the time that busy clinicians can devote to registry data collection is often limited.

Medical records also are a repository of clinician-derived data. Certain data about treatments, risk factors, and effect modifiers are often not consistently captured in medical records of any type, but where available, can be useful. Examples of such data that are difficult to find elsewhere include OTC medications, smoking and alcohol use, complementary and alternative medicines, and counseling activities by the clinician on lifestyle modifications. Medical records are often relied upon as a source of detailed clinical information for adjudication by external reviewers of medical diagnoses corresponding to study endpoints.

Electronic medical records, increasingly available, improve access to the data within medical records. The increasing use of electronic health records has facilitated the development of a number of registries within large health plans. Kaiser Permanente has created several registries of patients receiving total joint replacement, bariatric surgery, and nonsurgical conditions (e.g., diabetes), all of which rely heavily on existing electronic health record data. As discussed further in Chapter 15, the availability of medical records data in electronic format does not, by itself, guarantee consistency of terminology and coding.

Examples of other data sources include health insurance claims, pharmacy data, laboratory data, other registries, and national data sets, such as Medicare claims data and the National Death Index. These sources can be used to supplement registries with data that may otherwise be difficult to obtain, subject to recall bias, not collected because of loss to followup, or likely inaccurate by self-report (e.g., in those patients with diseases affecting recall, cognition, or mental status). See Table 6–1 (in Chapter 6) for more information on data sources.

5. Resources and Efficiency

Ideally, a study is designed to optimally answer a research question of interest and funded adequately to achieve the objectives based on the requirements of the design. Frequently, however, finite resources are available at the outset of a project that constrain the approaches that may be pursued. Often, through efficiencies in the selection of a study design and patient population (observational vs. RCT, case-control vs. prospective cohort), selection of data sources (e.g., medical-records–based studies vs. information collected directly from clinicians or patients), restriction of the number of study sites, or other approaches, studies may be planned that provide adequate evidence for addressing a research question, in spite of limited resources.

Section 6 below discusses how certain designs may be more efficient for addressing some research questions.

6. Study Designs for Registries

Although studies derived from registries are, by definition, observational studies, the framework for how the data will be analyzed drives the data collection and choices of patients for inclusion in the study.

The study models of case series, cohort, case-control, and case-cohort are commonly applied to registry data and are described briefly here. When case-control or case-cohort designs are applied to registry data, additional data may be collected to facilitate examination of questions that arise. Before adding new data elements, whether in a nested substudy or for a new objective, several of the steps outlined in Chapter 2, including assessing feasibility, considering the necessary scope and rigor, and evaluating the regulatory/ethical impact, should be undertaken. Other models that are also useful in some situations, but are not covered here, include: case-crossover studies, which are efficient designs for studying the effects of intermittent exposures (e.g., use of erectile dysfunction drugs) on conditions with sudden onset, and quasi-experimental studies or “pragmatic trials.” For example, in a pragmatic trial, providers may be randomized as to which intervention or quality improvement tools they use, but patients are observed without further intervention. Also, there has been recent interest in applying the concept of adaptive clinical trial design to registries. An adaptive design has been defined as a design that allows adaptations or modifications to some aspects of a clinical trial after its initiation without undermining the validity and integrity of the trial.³ While many long-term registries are modified after initiation, the more formal aspects of adaptive trial design have yet to be applied to registries and observational studies.

Determining what framework will be used to analyze the data is important in designing the registry and the registry data collection procedures. Readers are encouraged to consult textbooks of epidemiology and pharmacoepidemiology for more information. Many of the references in Chapters 13 and 18 relate to study design and analysis.

7. Choosing Patients for Study

The purpose of a registry is to provide information or describe events and patterns, and often to generate hypotheses about a specific patient population to whom study results are meant to apply. Studies can be conducted of people who share common characteristics, with or without the inclusion of comparison groups. For example, studies can be conducted of:

• People with a particular disease/outcome or condition. (These are focused on characteristics of the person.)

  –

  Examples include studies of the occurrence of cancer or rare diseases, pregnancy outcomes, and recruitment pools for clinical trials.

• Those with a particular exposure. (These exposures may be to a product, procedure, or other health service.)

  –

  Examples include general surveillance registries, pregnancy registries for particular drug exposures, and studies of exposure to medications and to devices such as stents.²¹ They also include studies of people who were treated under a quality improvement program, as well as studies of a particular exposure that requires controlled distribution, such as drugs with serious safety concerns (e.g., isotretinoin, clozapine, natalizumab [Tysabri^®]), where the participants in the registry are identified because of their participation in a controlled distribution/risk management program.

• Those who were part of a program evaluation, disease management effort, or quality improvement project.

  –

  An example is the evaluation of the effectiveness of evidence-based program guidelines on improving treatment.

8. Sampling

Various sampling strategies for patients and sites can be considered. Each of these has tradeoffs in terms of validity and information yield. The representativeness of the sample, with regard to the range of characteristics that are reflective of the broader target population, is often a consideration, but representativeness mainly affects generalizability rather than the internal validity of the results. Representativeness should be considered in terms of patients (e.g., men and women, children, the elderly, different racial or ethnic groups) and sites (academic medical centers, community practices). For sites (health care providers, hospitals, etc.), representativeness is often considered in terms of geography, practice size, and academic or private practice type. Reviewing and refining the research question can help researchers define an appropriate target population and a realistic strategy for subject selection.

To ensure that enough meaningful information will be available for analysis, registry studies often restrict eligibility for entry to individuals within a certain range of characteristics. Alternatively, they may use some form of sampling: random selection, systematic sampling, or a nonrandom approach. Often-used sampling strategies include the following:

• Probability sampling: Some form of random selection is used, wherein each person in the population must have a known (often equal) probability of being selected.^29-32

  –

  Census: A census sample includes every individual in a population or group (e.g., all known cases). A census is not feasible when the group is large relative to the costs of obtaining information from individuals.

  –

  Simple random sampling: The sample is selected in such a way that each person has the same probability of being sampled.

  –

  Stratified random sampling: The group from which the sample is to be taken is first stratified into subgroups on the basis of an important, related characteristic (e.g., age, parity, weight) so that each individual in a subgroup has the same probability of being included in the sample, but the probabilities for different subgroups or strata are different. Stratified random sampling ensures that the different categories of characteristics that are the basis of the strata are sufficiently represented in the sample. However, the resulting data must be analyzed using more complicated statistical procedures (such as Mantel-Haenszel) in which the stratification is taken into account.

  –

  Systematic sampling: Every nth person in a population is sampled.

  –

  Cluster (area) sampling: The population is divided into clusters, these clusters are randomly sampled, and then some or all patients within selected clusters are sampled. This technique is particularly useful in large geographic areas or when cluster-level interventions are being studied.

  –

  Multistage sampling: Multistage sampling can include any combination of the sampling techniques described above.

• Nonprobability sampling: Selection is systematic or haphazard but not random. The following sampling strategies affect the type of inferences that can be drawn; for example, it would be preferable to have a random sample if the goal were to estimate the prevalence of a condition in a population. However, systematic sampling of “typical” patients can generate useful data for many purposes, and is often used in situations where probability sampling is not feasible.³³

  –

  Case series or consecutive (quota) sampling: All consecutive eligible patients treated at a given practice or by a given clinician are enrolled until the enrollment target is reached. This approach is intended to reduce conscious or unconscious selection bias on the part of clinicians as to whom to enroll in the study, especially with regard to factors that may be related to prognosis.

  –

  Haphazard, convenience, volunteer, or judgmental sampling: This includes any sampling not involving a truly random mechanism. A hallmark of this form of sampling is that the probability that a given individual will be in the sample is unknown before sampling. The theoretical basis for statistical inference is lost, and the result is inevitably biased in unknown ways.

  –

  Modal instance: The most typical subject is sampled.

  –

  Purposive: Several predefined groups are deliberately sampled.

  –

  Expert: A panel of experts judges the representativeness of the sample or is the source that contributes subjects to a registry.

Individual matching of cases and controls is sometimes used as a sampling strategy for controls. Controls are matched with individual cases who have similar confounding factors, such as age, to reduce the effect of the confounding factors on the association being investigated.

Patients may be recruited in a fashion that allows for individual matching. For example, if a 69-year-old “case” participates in the registry, a control near in age will be sought. Individual matching for prospective recruitment is challenging and not customarily used. More often, matching is used to create subgroups for supplemental data collection for case-control studies and cohort studies when subjects are limited and/or stratification is unlikely to provide enough subjects in each stratum for meaningful evaluation.

A number of other sampling strategies have arisen from survey research (e.g., snowball, heterogeneity), but they are of less relevance to registries.

9. Registry Size and Duration

Precision in measurement and estimation corresponds to the reduction of random error; it can be improved by increasing the size of the study and modifying the design of the study to increase the efficiency with which information is obtained from a given number of subjects.²⁹

During the registry design stage, it is critical to explicitly state how large the registry will be, how long patients should be followed, and what the justifications are for these decisions. These decisions are based on the overall purpose of the registry. For example, in addressing specific questions of product safety or effectiveness, the desired level of precision to confirm or rule out the existence of an important effect should be specified, and ideally should be linked to policy or practice decisions that will be made based on the evidence. For registries with aims that are descriptive or hypothesis generating, study size may be arrived at through other considerations.

The duration of registry enrollment and followup should be determined both by required sample size (number of patients or person-years to achieve the desired power) and by time-related considerations. The induction period for some outcomes of interest must be considered, and sufficient followup time allowed for the exposure under study to have induced or promoted the outcome. Biological models of disease etiology and causation usually indicate the required time period of observation for an effect to become apparent. Calendar time may be a consideration in studies of changes in clinical practice or interventions that have a clear beginning and end. The need for evidence to inform policy may also determine a timeframe within which the evidence must be made available to decisionmakers.

A detailed discussion of the topic of sample size calculations for registries is provided in Appendix A. For present purposes it is sufficient to briefly describe some of the critical inputs to these calculations that must be provided by the registry developers:

• The expected timeframe of the registry and the time intervals at which analyses of registry data will be performed.
• Either the size of clinically important effects (e.g., minimum clinically important differences) or the desired precision associated with registry-based estimates.
• Whether or not the registry is intended to support regulatory decisionmaking. If the results from the registry will affect regulatory action—for example, the likelihood that a product may be pulled from the market—then the precision of the overall risk estimate is important, as is the necessity to predict and account for attrition.

In a classical calculation of sample size, the crucial inputs that must be provided by the investigators include either the size of clinically important effects or their required precision. For example, suppose that the primary goal of the registry is to compare surgical complication rates in general practice with those in randomized trials. The inputs to the power calculations would include the complication rates from the randomized trials (e.g., 4 percent) and the complication rate in general practice, which would reflect a meaningful departure from this rate (e.g., 6 percent). If, on the other hand, the goal of the registry is simply to track complication rates (and not to compare the registry with an external standard), then the investigators should specify the required width of the confidence interval associated with those rates. For example, in a large registry, the 95-percent confidence interval for a 5-percent complication rate might extend from 4.5 percent to 5.5 percent. If all of the points in this confidence interval lead to the same decision, then an interval of ±0.5 percent is considered sufficiently precise, and this is the input required for the estimation of sample size.

Specifying the above inputs to sample size calculations is a substantial matter and usually involves a combination of quantitative and qualitative reasoning. The issues involved in making this specification are essentially similar for registries and other study designs, though for registries designed to address multiple questions of interest, one or more primary objectives or endpoints must be selected that will drive the selection of a minimum sample size to meet those objectives.

Other considerations that should sometimes be taken into account when estimating sample sizes include—

• whether individual patients can be considered “independent,” or whether they share factors that would lead to correlation in measures between them;
• whether multiple comparisons are being made and subjected to statistical testing; and
• whether levels of expected attrition or lack of adherence to therapy may require a larger number of patients to achieve the desired number of person-years of followup or exposure.

In some cases, patients under study who share some group characteristics, such as patients treated by the same clinician or practice, or at the same institution, may not be entirely independent from one another with regard to some outcomes of interest or when studying a practice-level intervention. To the extent they are not independent, a measure of interdependence, the intraclass correlation (ICC), and so-called “design effect” must be considered in generating the overall sample size calculation. A reference addressing sample size considerations for a study incorporating a cluster-randomized intervention is provided.³⁴ A hierarchical or multilevel analysis may be required to account for one or more levels of “grouping” of individual patients, discussed further in Chapter 13, Section 5. One approach to addressing multiple comparisons in the surgical complication rate example above is to use control chart methodology, a statistical approach used in process measurement to examine the observed variability and determine whether out-of-control conditions are occurring. Control chart methodology is also used in sample size estimation, largely for studies with repeated measurements, to adjust the sample size as needed and therefore maintain reasonably precise estimates of confidence limits around the point estimate. Accordingly, for registries that involve ongoing evaluation, sample size per time interval could be determined by the precision associated with the related confidence interval, and decision rules for identifying problems could then be based on control chart methodology.

Although most of the emphasis in estimating study size requirements is focused on patients, it is equally important to consider the number of sites needed to recruit and retain enough patients to achieve a reasonably informative number of person-years for analysis. Many factors are involved in estimating the number of sites needed for a given study, including the number of eligible patients seen in a given practice during the relevant time period, desired representativeness of sites with regard to geography, practice size, or other features, and the timeframe within which study results are required, which may also limit the timeframe for patient recruitment.

In summary, the aims of a registry, the desired precision of information sought, and the hypotheses to be tested, if any, determine the process and inputs for arriving at a target sample size and specifying the duration of followup.

Registries with mainly descriptive aims, or those that provide quality metrics for clinicians or medical centers, may not require the choice of a target sample size to be arrived at through power calculations. In either case, the costs of obtaining study data, in monetary terms and in terms of researcher, clinician, and patient time and effort, may set upper as well as lower limits on study size. Limits to study budgets and the number of sites and patients that could be recruited may be apparent at the outset of the study. However, an underpowered study involving substantial data collection that is ultimately unable to satisfactorily answer the research question(s) may prove to be a waste of finite monetary as well as human resources that could better be applied elsewhere.

10. Internal and External Validity

The potential for bias refers to opportunities for systematic errors to influence the results. Internal validity is the extent to which study results are free from bias, and the reported association between exposure and outcome is not due to unmeasured or uncontrolled-for variables. Generalizability, also known as external validity, is a concept that refers to the utility of the inferences for the broader population that the study subjects are intended to represent. In considering potential biases and generalizability, we discuss the differences between RCTs and registries, since these are the two principal approaches to conducting clinically relevant prospective research.

The strong internal validity that earns RCTs high grades for evidence comes largely from the randomization of exposures that helps ensure that the groups receiving the different treatments are similar in all measured or unmeasured characteristics, and that, therefore, any differences in outcome (beyond those attributable to chance) can be reasonably attributed to differences in the efficacy or safety of the treatments. However, it is worth noting that RCTs are not without their own biases, as illustrated by the “intent-to-treat” analytic approach, in which people are considered to have used the assigned treatment, regardless of actual compliance. The intent-to-treat analyses can minimize a real difference—generating a distortion known as “bias toward the null”—by including the experience of people who did not adhere to the recommended study product along with those who did.

Another principal difference between registries and RCTs is that RCTs are often focused on a relatively homogeneous pool of patients from which significant numbers of patients are purposefully excluded at the cost of external validity—that is, generalizability to the target population of disease sufferers. Registries, in contrast, usually focus on generalizability so that their population will be representative and relevant to decisionmakers.

11. Summary

In summary, the key points to consider in designing a registry include study design, data sources, patient selection, comparison groups, sampling strategies, and considerations of possible sources of bias and ways to address them, to the extent that is practical and achievable.

Case Examples for Chapter 3

References for Chapter 3

1.

Strom BL. Pharmacoepidemiology. 3rd ed. Chichester, England: John Wiley; 2000.

2.

Gliklich R. A New Framework For Comprehensive Evidence Development. IN VIVO. 2012 Oct 22 [June 17, 2013]; http://www​.elsevierbi​.com/publications/in-vivo​/30/9/a-new-framework-for-comprehensive-evidence-development.

3.

Chow SC, Chang M, Pong A. Statistical consideration of adaptive methods in clinical development. J Biopharm Stat. 2005;15(4):575–91. [PubMed: 16022164]

4.

Farrington CP. Relative incidence estimation from case series for vaccine safety evaluation. Biometrics. 1995 Mar;51(1):228–35. [PubMed: 7766778]

5.

Ryan PB, Powell GE, Pattishall EN, et al. Performance of screening multiple observational databases for active drug safety surveillance. Providence, RI: International Society of Pharmacoepidemiology; 2009.

6.

Norén GN, Hopstadius J, Bate A, et al. Temporal pattern discovery in longitudinal electronic patient records. Data Min Knowl Discov. 2010;20(3):361–87.

7.

Travis LB, Rabkin CS, Brown LM, et al. Cancer survivorship—genetic susceptibility and second primary cancers: research strategies and recommendations. J Natl Cancer Inst. 2006 Jan 4;98(1):15–25. [PubMed: 16391368]

8.

Sackett DL, Haynes RB, Tugwell P. Clinical epidemiology. Boston: Little, Brown and Company; 1985. p. 228.

9.

Hennekens CH, Buring JE. Epidemiology in medicine. 1st ed. Boston: Little, Brown and Company; 1987.

10.

Schoebel FC, Gradaus F, Ivens K, et al. Restenosis after elective coronary balloon angioplasty in patients with end stage renal disease: a case-control study using quantitative coronary angiography. Heart. 1997 Oct;78(4):337–42. [PMC free article: PMC1892250] [PubMed: 9404246]

11.

Rothman K, Greenland S. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 1998. pp. 175–9.

12.

Oral contraceptive use and the risk of endometrial cancer. The Centers for Disease Control Cancer and Steroid Hormone Study. JAMA. 1983 Mar 25;249(12):1600–4. [PubMed: 6338265]

13.

Oral contraceptive use and the risk of ovarian cancer. The Centers for Disease Control Cancer and Steroid Hormone Study. JAMA. 1983 Mar 25;249(12):1596–9. [PubMed: 6338264]

14.

Long-term oral contraceptive use and the risk of breast cancer. The Centers for Disease Control Cancer and Steroid Hormone Study. JAMA. 1983 Mar 25;249(12):1591–5. [PubMed: 6338262]

15.

Speck CE, Kukull WA, Brenner DE, et al. History of depression as a risk factor for Alzheimer's disease. Epidemiology. 1995 Jul;6(4):366–9. [PubMed: 7548342]

16.

Wacholder S, McLaughlin JK, Silverman DT, et al. Selection of controls in case-control studies. I. Principles. Am J Epidemiol. 1992 May 1;135(9):1019–28. [PubMed: 1595688]

17.

Wacholder S, Silverman DT, McLaughlin JK, et al. Selection of controls in case-control studies. II. Types of controls. Am J Epidemiol. 1992 May 1;135(9):1029–41. [PubMed: 1595689]

18.

Wacholder S, Silverman DT, McLaughlin JK, et al. Selection of controls in case-control studies. III. Design options. Am J Epidemiol. 1992 May 1;135(9):1042–50. [PubMed: 1595690]

19.

Rothman K, Greenland S. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 1998. p. 108.

20.

Vigneswaran R, Aitchison SJ, McDonald HM, et al. Cerebral palsy and placental infection: a case-cohort study. BMC Pregnancy Childbirth. 2004 Jan 27;4(1):1. [PMC free article: PMC343280] [PubMed: 15005809]

21.

Ong AT, Daemen J, van Hout BA, et al. Cost-effectiveness of the unrestricted use of sirolimus-eluting stents vs. bare metal stents at 1 and 2-year follow-up: results from the RESEARCH Registry. Eur Heart J. 2006 Dec;27(24):2996–3003. [PubMed: 17114234]

22.

Hulley SB, Cumming SR. Designing clinical research. Baltimore: Williams & Wilkins; 1988.

23.

Hunter D. First, gather the data. N Engl J Med. 2006 Jan 26;354(4):329–31. [PubMed: 16436764]

24.

Mangano DT, Tudor IC, Dietzel C, et al. The risk associated with aprotinin in cardiac surgery. N Engl J Med. 2006 Jan 26;354(4):353–65. [PubMed: 16436767]

25.

National Cancer Institute. Surveillance Epidemiology and End Results. [August 27, 2012]. http://seer​.cancer.gov.

26.

Metropolitan Atlanta Congenital Defects Program (MACDP). National Center on Birth Defects and Developmental Disabilities. Centers for Disease Control and Prevention; [August 27, 2012]. http://www​.cdc.gov/ncbddd​/birthdefects/MACDP.html. [PubMed: 17109399]

27.

Chen E, Sapirstein W, Ahn C, et al. FDA perspective on clinical trial design for cardiovascular devices. Ann Thorac Surg. 2006 Sep;82(3):773–5. [PubMed: 16928481]

28.

U.S. Food and Drug Administration; Center for Devices and Radiological Health. The Least Burdensome Provisions of the FDA Modernization Act of 1997; Concept and Principles: Final Guidance for FDA and Industry. [August 14, 2012]. Document issued October 4, 2002. http://www​.fda.gov/MedicalDevices​/DeviceRegulationandGuidance​/GuidanceDocuments/ucm085994.htm#h3.

29.

Cochran WG. Sampling Techniques. 3rd ed. New York: John Wiley & Sons; 1977.

30.

Lohr SL. Sampling: Design and Analysis. Boston: Duxbury; 1999.

31.

Sudman S. Applied Sampling. New York: Academic Press; 1976.

32.

Henry GT. Practical Sampling. Newbury Park, CA: Sage; 1990.

33.

Rothman K, Greenland S. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 1998. p. 116.

34.

Raudenbush SW. Statistical analysis and optimal design for cluster randomized trials. Psychological Methods. 1997;2(2):173–85.

35.

Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000 Jun 22;342(25):1887–92. [PMC free article: PMC1557642] [PubMed: 10861325]

36.

Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000 Jun 22;342(25):1878–86. [PubMed: 10861324]

37.

Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996 May 11;312(7040):1215–8. [PMC free article: PMC2350940] [PubMed: 8634569]

38.

Rothman K. Modern Epidemiology. Boston: Little Brown and Company; 1986. p. 83.

39.

Petri H, Urquhart J. Channeling bias in the interpretation of drug effects. Stat Med. 1991 Apr;10(4):577–81. [PubMed: 2057656]

40.

Johnston SC. Identifying confounding by indication through blinded prospective review. Am J Epidemiol. 2001 Aug 1;154(3):276–84. [PubMed: 11479193]

41.

Sturmer T, Joshi M, Glynn RJ, et al. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol. 2006 May;59(5):437–47. [PMC free article: PMC1448214] [PubMed: 16632131]

42.

Glynn RJ, Schneeweiss S, Sturmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. 2006 Mar;98(3):253–9. [PMC free article: PMC1790968] [PubMed: 16611199]

43.

Loughlin J, Seeger JD, Eng PM, et al. Risk of hyperkalemia in women taking ethinylestradiol/ drospirenone and other oral contraceptives. Contraception. 2008 Nov;78(5):377–83. [PubMed: 18929734]

44.

Brookhart M. Alan. Instrumental Variables for Comparative Effectiveness Research: A Review of Applications. Rockville, MD: Agency for Healthcare Research and Quality; Jan, 2009. [August 14, 2012]. Slide Presentation from the AHRQ 2008 Annual Conference (Text Version) http://www​.ahrq.gov/about​/annualmtg08/090908slides​/Brookhart.htm.

45.

Evans WN, Ringel JS. Can higher cigarette taxes improve birth outcomes? J Public Economics. 1999;72(1):135–54.

46.

Schneeweiss S, Seeger JD, Landon J, et al. Aprotinin during coronary-artery bypass grafting and risk of death. N Engl J Med. 2008 Feb 21;358(8):771–83. [PubMed: 18287600]

47.

Bosco JL, Silliman RA, Thwin SS, et al. A most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies. J Clin Epidemiol. 2010 Jan;63(1):64–74. [PMC free article: PMC2789188] [PubMed: 19457638]

48.

Ray WA. Evaluating medication effects outside of clinical trials: new-user designs. Am J Epidemiol. 2003 Nov 1;158(9):915–20. [PubMed: 14585769]

49.

Kristman V, Manno M, Cote P. Loss to follow-up in cohort studies: how much is too much? Eur J Epidemiol. 2004;19(8):751–60. [PubMed: 15469032]

50.

Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data. Springer; 2009.