Purpose and Limitations of Evidence Reports

Systematic reviews, or evidence reports, are the building blocks underlying evidence-based practice. An evidence report focuses attention on the strength and limits of evidence from published studies about the effectiveness of a clinical intervention. The development of an evidence report begins with a careful formulation of the problem. The goal is to select questions that are important to patients and clinicians, then to examine how well the scientific literature answers those questions. Terms commonly used in systematic reviews, such as statistical terms, are provided in Appendix A and are defined as they apply to reports produced by the Drug Effectiveness Review Project.

An evidence report emphasizes the patient’s perspective in the choice of outcome measures. Studies that measure health outcomes (events or conditions that the patient can feel, such as quality of life, functional status, and fractures) are emphasized over studies of intermediate outcomes (such as changes in bone density). Such a report also emphasizes measures that are easily interpreted in a clinical context. Specifically, measures of absolute risk or the probability of disease are preferred to measures such as relative risk. The difference in absolute risk between interventions is dependent on the numbers of events in both groups, such that the difference (absolute risk reduction) is smaller when there are fewer events. In contrast, the difference in relative risk is fairly constant across groups with different baseline risk for the event, such that the difference (relative risk reduction) is similar across these groups. Relative risk reduction is often more impressive than the absolute risk reduction. Another measure useful in applying the results of a study is the number needed to treat (or harm), the NNT (or NNH). The NNT represents the number of patients who would have to be treated with an intervention for 1 additional patient to benefit (experience a positive outcome or avoid a negative outcome). The absolute risk reduction is used to calculate the NNT.

An evidence report also emphasizes the quality of the evidence, giving more weight to studies that meet high methodological standards that reduce the likelihood of biased results. In general, for questions about the relative benefits of a drug, the results of well-done, randomized controlled trials are regarded as better evidence than results of cohort, case-control, or cross-sectional studies. In turn, these studies are considered better evidence than uncontrolled trials or case series. For questions about tolerability and harms, controlled trials typically provide limited information. For these questions, observational study designs may provide important information that is not available from trials. Within this hierarchy, cohort designs are preferred when well conducted and assessing a relatively common outcome. Case control studies are preferred only when the outcome measure is rare, and the study is well conducted.

An evidence report pays particular attention to the generalizability of efficacy studies performed in controlled or academic settings. Efficacy studies provide the best information about how a drug performs in a controlled setting that allows for better control over potential confounding factors and bias. However, the results of efficacy studies are not always applicable to many, or to most, patients seen in everyday practice. This is because most efficacy studies use strict eligibility criteria that may exclude patients based on their age, sex, medication compliance, or severity of illness. For many drug classes, including antipsychotics, unstable or severely impaired patients are often excluded from trials. Often, efficacy studies also exclude patients who have comorbiddiseases, meaning diseases other than the one under study. Efficacy studies may also use dosing regimens and follow-up protocols that may be impractical in other practice settings. They often restrict options, such as combining therapies or switching drugs that are of value in actual practice. They often examine the short-term effects of drugs that in practice are used for much longer periods of time. Finally, efficacy studies tend to use objective measures of effects that do not capture all of the benefits and harms of a drug or do not reflect the outcomes that are most important to patients and their families.

An evidence report also highlights studies that reflect actual clinical effectiveness in unselected patients and community practice settings. Effectiveness studies conducted in primary care or office-based settings use less stringent eligibility criteria, assess health outcomes, and have longer follow-up periods than most efficacy studies. The results of effectiveness studies are more applicable to the “average” patient than results from highly selected populations in efficacy studies. Examples of effectiveness outcomes include quality of life, hospitalizations, and the ability to work or function in social activities. These outcomes are more important to patients, family, and care providers than surrogate or intermediate measures such as scores based on psychometric scales.

Efficacy and effectiveness studies overlap. For example, a study might use very narrow inclusion criteria like an efficacy study, but, like an effectiveness study, might examine flexible dosing regimens, have a long follow-up period, and measure quality of life and functional outcomes. For this report we sought evidence about outcomes that are important to patients and would normally be considered appropriate for an effectiveness study. However, many of the studies that reported these outcomes were short-term and used strict inclusion criteria to select eligible patients. For these reasons, it is neither possible nor desirable to exclude evidence based on these characteristics. Labeling each study as an efficacy or effectiveness study, while convenient, is of limited value; it is more useful to consider whether the patient population, interventions, time frame, and outcomes are relevant to one’s practice, or, in the clinical setting, how relevant they are to a particular patient.

Studies across the continuum from efficacy to effectiveness can be useful in comparing the clinical value of different drugs. Effectiveness studies are more applicable to practice, but efficacy studies are a useful scientific standard to determine whether the characteristics of different drugs are related to their effects on disease. An evidence report reviews the efficacy data thoroughly to ensure that decision-makers can assess the scope, quality, and relevance of the available data. This thoroughness is not intended to obscure the fact that efficacy data, no matter how much there is of it, may have limited applicability to practice. Clinicians can judge the relevance of the study results to their practice and should note where there are gaps in the available scientific information.

Unfortunately, for many drugs, there are few or no effectiveness studies and many efficacy studies. As a result, clinicians must make decisions about treatment for many patients who would not have been included in controlled trials and for whom the effectiveness and tolerability of the different drugs are uncertain. An evidence report indicates whether or not there is evidence that drugs differ in their effects in various subgroups of patients, but it does not attempt to set a standard for how results of controlled trials should be applied to patients who would not have been eligible for them. With or without an evidence report, these are decisions that must be informed by clinical judgment.

In the context of developing recommendations for practice, evidence reports are useful because they define the strengths and limits of the evidence, clarifying whether assertions about the value of the intervention are based on strong evidence from clinical studies. By themselves, they do not tell you what to do. Judgment, reasoning, and applying one’s values under conditions of uncertainty must also play a role in decision making. Users of an evidence report must also keep in mind that not proven does not mean proven not; that is, if the evidence supporting an assertion is insufficient, it does not mean the assertion is not true. The quality of the evidence on effectiveness is a key component, but not the only component, in making decisions about clinical policies. Additional criteria include acceptability to physicians or patients, the potential for unrecognized harms, the applicability of the evidence to practice, and consideration of equity and justice.

Scope and Key Questions

The participating organizations of the Drug Effectiveness Review Project are responsible for ensuring that the scope of the review reflects the populations, drugs, and outcome measures of interest to their constituencies. Initially, the Oregon Evidence-based Practice Center wrote preliminary key questions, identifying the populations, interventions, and outcomes of interest, and based on these, the eligibility criteria for studies. These were reviewed, revised, and approved by representatives of organizations participating in the Drug Effectiveness Review Project. It is the representatives' responsibility to ensure that the questions reflect public input or input from their members. The participating organizations approved the following key questions to guide this review.

Key Question 1. For adult patients with hypertension, angina, coronary artery bypass graft, recent myocardial infarction, heart failure, atrial arrhythmia, migraine or bleeding esophageal varices, do beta blocker drugs differ in effectiveness/efficacy?

Key Question 2. For adult patients with hypertension, angina, coronary artery bypass graft, recent myocardial infarction, heart failure, atrial arrhythmia, migraine prophylaxis or bleeding esophageal varices, do beta blocker drugs differ in harms?

Key Question 3. Are there subgroups of patients based on demographics (age, racial groups, gender), other medications (drug-drug interactions), or co-morbidities (drug-disease interactions) for which one beta blocker is more effective or associated with fewer adverse effects?

This review includes beta blockers that are available in the United States in an oral form and are indicated for hypertension. We excluded esmolol, an ultra-short acting beta blocker available only in intravenous form. Esmolol is used primarily as an antiarrhythmic drug for intraoperative and other acute arrhythmias. We also excluded sotalol, a nonselective beta blocker with Class III antiarrhythmic activity that is used exclusively for arrhythmias. Beta blockers that are unavailable in the United States are bopindolol, bucindolol, medroxalol, and oxprenolol.