Topic Development

The topic for this report was nominated in a public process. At the beginning of the project, we recruited a panel of key informants to give input on key steps including the selection and refinement of the questions to be examined. The panel included internal experts from the Johns Hopkins University with expertise in evaluating the efficacy and safety of immunotherapy and external experts with expertise in immunotherapy research and patient care.

In preparation for this report, we reviewed existing systematic reviews on this topic as well as guidelines prepared by key professional societies about the use of these therapies. With input from the key informants, staff of AHRQ, and the Scientific Resources Center, we developed the KQs. Our draft KQs were posted on AHRQ’s website for public comment in April 2011. We then refined the KQs based on feedback received.

The final KQs focus on the comparisons of the methods of immunotherapy delivery, their ability to affect intermediate outcomes, long-term clinical outcomes, and adverse effects. We drafted a protocol to address these KQs and then recruited a panel of technical experts, which included experts on the treatment of allergies on the adult and pediatric population, including asthma experts. With input from the technical expert panel and representatives from AHRQ, we finalized the protocol.

Search Strategy

We searched the following databases for primary studies for the periods in parentheses: MEDLINE^® (from 1950 to May 21 2012), Embase (from 1947 to May 21 2012), the Cochrane Central Register of Controlled Trials (to May 21 2012), and LILACS (Latin American and Caribbean Health Sciences Literature, from 1982 to May 21 2012). We developed a search strategy for MEDLINE, accessed via PubMed, based on an analysis of the medical subject headings (MeSH), terms, and text words of key articles identified a priori (Appendix A). We also reviewed the reference lists of each included articles and relevant review articles.

To identify additional studies, we reviewed public registries of clinical trials, including the International Clinical Trials Registry Platform Search Portal (http://apps.who.int/trialsearch/default.aspx) and ClinicalTrials.gov (www.clinicaltrials.gov). We also assessed medical and statistical reviews, as well as the FDA status of the included medications, using the Food and Drug Administration website.

The results of the searches were downloaded and imported into ProCite^® version 5 (ISI Research Soft, Carlsbad, CA). We scanned for exact article duplicates; author/title duplicates, and title duplicates using the duplication check feature in ProCite. From ProCite, the articles were uploaded to DistillerSR (Evidence Partners, Ottawa, Ontario, Canada), a Web-based software package developed for systematic review and data management. This database was used to track the search results at the levels of abstract review, article inclusion/exclusion, and data abstraction.

We requested Scientific Information Packets from the relevant pharmaceutical companies so as to be able to include gray literature in this review.

Study Selection

The abstract review phase was designed to identify randomized controlled trials (RCTs) reporting on the effects of SIT on intermediate outcomes, long-term clinical outcomes, and/or adverse events and side effects (Appendix B). We included only articles published in English due to volume of literature and lack of resources to translate all the languages encountered. Abstracts were reviewed independently by two investigators and were excluded if both investigators agreed that the article met one or more of the exclusion criteria (Table 1). Differences between investigators regarding abstract inclusion or exclusion were resolved through consensus adjudication.

Table 1

Study inclusion and exclusion criteria.

Articles promoted on the basis of abstract review underwent another independent parallel review to determine if they should be included for data abstraction (Appendix B). Differences regarding article inclusion were resolved through consensus adjudication. A third reviewer audited a random sample of abstract and article reviews to ensure consistency in the reviewing process.

Studies utilizing sublingual formulations not currently available or in which similar off label use allergens are not available in the United States such as sublingual tablets, were not included in this review. We also excluded articles in which oral immunotherapy was immediately swallowed without prolonged mucosal contact, as this type of immunotherapy is not currently in clinical use.

All of the articles had to meet four basic criteria to be included: the allergic diagnosis had to be confirmed, the study had to include a relevant comparison group, the dose of allergen had to be specified, and the study had to report the outcomes of interest.

The studies compared the outcomes of patients receiving immunotherapy to the outcomes of patients that did not receive immunotherapy. The comparator arms sometimes included administration of a placebo and uniformly included pharmacotherapy for symptom control, which can be considered to be usual care. The majority of immunotherapy arms also permitted concurrent use of pharmacotherapy.

In this review, multiple allergen immunotherapy was defined as the use of extracts containing more than one allergen species, including cross-reacting allergens. Single allergen immunotherapy was defined by the use of a single allergen species, and not by a class of allergens.

Allergists may apply different definitions of single and multiple allergen immunotherapies to our findings. Multiple allergen immunotherapies can be defined as the use of extracts containing more than one allergen class, whereas single allergen immunotherapy can refer to the use of closely related allergens within the same class. For example, a study using a grass mix allergen (or tree mix, or 2 dust mite species) could be considered a single allergen study, whereas a multiple allergen study could use different classes of allergens, such as tree and grass.

Data Abstraction

We used a systematic approach for extracting data to minimize the risk of bias in this process. By creating standardized forms for data extraction, which were pilot tested, we sought to maximize consistency in identifying all pertinent data available for synthesis. Each article underwent double review by study investigators for data abstraction. The second reviewer confirmed the first reviewer’s data abstraction for completeness and accuracy. Reviewer pairs were formed to assure clinical and methodological expertise. A third reviewer re-reviewed a random sample of articles by the first two reviewers to ensure consistency in the data abstraction of the articles. Reviewers were not masked to the articles’ authors, institution, or journal. In most instances, data were abstracted from the text or tables in the article. If possible, relevant data were also abstracted from figures. Differences in opinion were resolved through consensus adjudication and in difficult cases, during team meetings.

For all articles, reviewers extracted information on general study characteristics (for example, study design, study period, and followup); study participants (for example, age, sex, race, disease, inclusion criteria, allergens, and duration of disease); interventions (for example, doses, frequency of use, and duration of use); primary and secondary outcome measures, their the method of ascertainment, and the results of each outcome; and safety (Appendix B). For studies that recorded outcomes at multiple time points, we used the outcome data from the final time point reported. However, some studies treated and assessed subjects for only one season; in these single season studies, the values reported at peak pollen seasons were used when available.

All information from the article review process was entered into the DistillerSR database by the individual completing the review. Reviewers entered comments into the system whenever applicable. The DistillerSR database was used to maintain and clean the data, as well as to create detailed evidence tables and summary tables.

Quality Assessment

Two reviewers independently assessed the risk of bias in each article and came to consensus about the overall rating. We used a modification of the Cochrane Collaboration Tool for Assessing Risk of Bias from the Cochrane Handbook for Systematic Reviews of Interventions.²⁹ This tool was used to assess potential sources of bias:

1. Was there random allocation of subjects?
2. Was the allocation scheme concealed?
3. Was the intervention concealed from study personnel and participants?
4. Was incomplete data adequately addressed?
5. Were there other important sources of bias?

We did not assess selective outcome reporting in this body of literature. We did, however, assess a sixth item: the participation of the sponsor company in the study design and interpretation.

For each bias category, reviewers entered “Yes” if item posed a low risk of bias, “No” if item posed a high risk of bias, or “Unclear” (Appendix C).

• Good (low risk of bias). 0–1 point. These studies had the least bias, and the results were considered valid. These studies adhered to the commonly held concepts of high quality, including the following: a formal randomized controlled design; a clear description of the population, setting, interventions, and comparison groups; appropriate measurement of outcomes; appropriate statistical and analytic methods and reporting; no reporting errors; a low dropout rate; and clear reporting of dropouts.
• Fair (moderate risk of bias). 2–3 points. These studies were susceptible to some bias, but not enough to invalidate the results. They did not meet all the criteria required for a rating of good quality because they had some deficiencies, but no flaw was likely to cause major bias. The study may have been missing information, making it difficult to assess limitations and potential problems.
• Poor (high risk of bias). 4–6 points. These studies had significant flaws that might have invalidated the results. They had serious errors in design, analysis, or reporting; large amounts of missing information; or discrepancies in reporting.

We reviewed all of the studies that had only one point in the overall quality assessment and made some reassignments. Studies remained in the Good (low risk of bias) category if the single point was due to sponsorship or “other sources of bias”; studies were assigned to the Fair (moderate risk of bias) category if the single point came from lack of allocation concealment, lack of blinding or incomplete data reporting.

Data Analysis and Synthesis

We distributed the studies by intervention, disease, and allergen KQ following the following diagram, and addressed the KQs within each intervention and disease strata (Figure 2).

Figure 2

Algorithm for the approach and classification of the studies. SCIT = subcutaneous immunotherapy; SIT = allergen specific immunotherapy; SLIT = sublingual immunotherapy

We created a set of detailed evidence tables containing information extracted from eligible studies and stratified the tables according to KQ. Once these evidence tables were created, we rechecked selected data elements against the original articles. If there was a discrepancy between the data abstracted and the data appearing in the article, this discrepancy was brought to the attention of the investigator in charge of the specific dataset and the data were corrected in the final evidence tables. Given the substantial heterogeneity between studies and the lack of reporting of measures of variability, we did not quantitatively pool the data.

We summarized the safety of sublingual immunotherapy in the treatment of allergic rhinoconjunctivitis and/or asthma by abstracting the harms or adverse events reported in the included studies. The adverse events recorded with sublingual immunotherapy were divided into two general categories. Local reactions are reactions that occur at the site of introduction of allergen. In the case of sublingual immunotherapy, these are reactions that occur in the oral cavity, such as mouth irritation, itching, swelling, and pain. The reactions may or may not require treatment and can range from mild to severe. Systemic reactions are allergic reactions that occur distant to the site of introduction of the allergen and can include any system of the body: cutaneous, ocular, gastrointestinal, or respiratory. These reactions may or may not require treatment, and some may require hospitalization. Severity can range from mild to life-threatening. The most severe potential systemic reactions with allergen-specific immunotherapy include anaphylaxis and death.

Studies used different methods for reporting safety data. The two most common methods were number of patients experiencing adverse events and number of adverse events experienced throughout study period. Due to the heterogeneity observed in the different studies, the safety outcomes are presented only descriptively.

Data Entry and Quality Control

Each data element was reviewed by at least two reviewers. The second reviewers were generally more experienced members of the research team. In addition, two additional investigators audited a random sample of the reviews to identify any problems with data abstraction. If problems were recognized in a reviewer’s data abstraction, the problems were discussed at a meeting with the reviewers. In addition, research assistants used a system of random data checks to assure data abstraction accuracy.

Rating Body of Evidence

At the completion of our review, we graded the quantity, quality, and consistency of the best available evidence addressing the three KQs by adapting by the Grading of Recommendation Assessment, Development and Evaluation (GRADE) Working Group, adapted by AHRQ in the “Methods Guide for Effectiveness and Comparative Effectiveness Reviews” (www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=328&pageaction=displayproduct) and published in the Journal of Clinical Epidemiology.^30,31

We applied evidence grades to the collection of trials for each comparison and for each outcome. We found that some articles reported only the post- to pre- comparisons within the intervention arm. We show these results in our evidence tables and summary tables, however, those results did not contribute to the evidence grades as this is a less strong design than the head-to-head comparisons. In our grade assignments, we considered the limitations of each individual study’s quality (using the risk of bias classification), the consistency of the direction of the effect across studies, the directness of the body of evidence to the question of interest, and the magnitude of the effects reported across trials. We could not comment on the precision of the effect sizes as there were seldom measures of variance within the individual studies. We did not use the reported statistical significance of the differences between groups to grade the evidence as this was not consistently reported. We could not generate confidence intervals for these data as these were largely continuous outcomes. We calculated the percent change in outcomes in the intervention arm, and also the percent change in the comparator arm; the magnitude of effect was based on the difference between comparators.

There is no clear consensus on what is considered a clinically relevant improvement in symptoms. While some clinicians may suggest that a 15 percent change could reflect real and significant improvement in symptoms in some patients, Canonica et al state that “the minimal clinically relevant efficacy should be at least 20 percent higher than placebo.”¹⁰ We would expect less difference in symptom improvement when comparing immunotherapy to medications. Our systematic review included both studies using placebo and other comparators (such as medications). We chose to classify magnitude of effect as weak if there was less than a 15 percent difference in percent change between the SIT group and comparator arm; a 15 to 40 percent difference was called moderate, and greater than 40 percent was considered a strong effect. We applied this scheme to all graded outcomes in this review. We did not grade the evidence for indirect outcomes such as pulmonary function testing and provocational studies.

The investigator responsible for each section assigned an evidence grade for each disease (asthma, allergic rhinitis, and rhinoconjunctivitis) and each treatment comparison. The team reviewed these and came to a consensus. We assigned evidence grades as:

1. High grade (indicating high confidence that the evidence reflects the true effect and further research is very unlikely to change our confidence in the estimate of the effect);
2. Moderate grade (indicating moderate confidence that the evidence reflects the true effect and future research may change our confidence in the estimate of the effect and may change the estimate);
3. Low grade (indicating low confidence that the evidence reflects the true effect and further research is likely to change our confidence in the estimate of the effect and is likely to change the estimate); and
4. Insufficient (evidence is unavailable or no relevant trials).

We adhered to the following system to assign the overall grade of evidence for each outcome. High grade evidence is at least 2 trials having low risk of bias, at least 1 of which has a strong magnitude of effect and the overall body of evidence is largely consistent. Moderate grade evidence is 1 trial having a low risk of bias with a strong magnitude of effect; or 2 or more trials with medium risk of bias having strong magnitudes of effect, or 1 trial having low risk of bias with moderate magnitude of effect plus 1 trial having medium risk of bias with strong magnitude of effect and an overall body of evidence that is largely consistent. Low grade evidence was assigned if there was evidence but it did not meet the criteria for the above categories. Evidence was insufficient if there were no relevant trials or data were insufficient.

If the evidence did not meet the criteria to be rated as high then it was graded as moderate IF it met criteria for moderate, if not then it was graded as low. A body of evidence was considered consistent if the direction of effect was the same for all studies for a given comparison and outcome.

The safety data reported in this systematic review include only events reported in RCTs. Evidence grades on the safety of SIT using only this data would be invalid since the grades would not be based on the entirety of the evidence, as safety events are more completely captured in observational studies. Given this, we chose not to grade the safety data. Additionally, the lack of consistency on the reporting of adverse events and the differences in the severity grading systems made the safety data difficult to synthesize.

Applicability

Throughout the report, we discuss the applicability of the results as the degree to which the study population, interventions, outcomes, and settings are typical of treatment of individuals with allergic rhinitis and asthma in usual care settings (for example, outpatient treatment by internists, family physicians, pediatricians, allergists, and otolaryngologists).

Peer Review and Public Commentary

A draft of the evidence report was reviewed by the peer reviewers, AHRQ representatives, and the Eisenberg Center.