Topic Refinement and Review Protocol

The initial Key Questions were provided by the National Highway Traffic Safety Administration (NHTSA). The Key Questions were further developed and the final protocol developed with additional input from NHTSA and a Technical Expert Panel (TEP) convened for this report. The TEP consisted of 10 experts in addiction medicine, pain medicine, emergency medical services (EMS), emergency medicine, toxicology, pharmacy, epidemiology of opioid overdose, opioid overdose research, and public policy. TEP members disclosed financial and other conflicts of interest prior to participation. The AHRQ Task Order Officer and the investigators reviewed the disclosures and determined that the TEP members had no conflicts of interest that precluded participation.

The final protocol was posted on the AHRQ Web site on November 30, 2016 at: https://www.effectivehealthcare.ahrq.gov/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=2360. The protocol was also registered in the PROSPERO international database of prospectively registered systematic reviews (registration number CRD42016053891). Two modifications were made to the protocol after posting: we expanded the inclusion criteria for Key Question 1 to include studies performed in an emergency department (ED) setting and we expanded the inclusion criteria for Key Question 4 to include uncontrolled longitudinal studies of patients successfully treated for suspected opioid overdose in the field who were not transported to a health care facility.

Literature Search Strategy

Inclusion and Exclusion Criteria

We developed preestablished criteria for inclusion and exclusion of studies based on the Key Questions and the populations, interventions, comparators, outcomes, timing, types of studies, and setting (PICOTS) approach, in accordance with the AHRQ EPC Methods Guide for Effectiveness and Comparative Effectiveness Reviews.⁶⁰ Inclusion and exclusion criteria are described below.

Abstracts were reviewed by two investigators, and all citations deemed potentially appropriate for inclusion by at least one of the reviewers were retrieved for full-text review. Two investigators then independently reviewed all full-text articles for final inclusion. Investigators did not review, assess, or screen papers that they had authored. Inclusion was restricted to English-language articles. Discrepancies were resolved by discussion and consensus, with a third investigator to resolve disagreements if necessary.

A list of the included studies is shown in Appendix B; a list of excluded studies and primary reasons for exclusion is shown in Appendix C.

Interventions

• For Key Questions 1–3: See Table 1 for included naloxone formulations.
• Potential modifiers of interventions: Based on training and background of the person administering naloxone
• For Key Question 4: Transport to health care facility
• Exclude: Naloxone in combination with other medications (e.g., buprenorphine/naloxone)

Table 1

Naloxone: Dose and route of administration.

Data Extraction

For each study that met inclusion criteria, a single investigator abstracted information on study design, year, setting (field or ED), country, sample size, eligibility criteria, population and clinical characteristics (age, sex, race, type of opioid involved in overdose, dose of opioid involved in overdose, presence of other drugs or substances contributing to overdose, estimated time since overdose, concomitant psychiatric comorbidities, prior overdose episodes), intervention characteristics (route of administration, dose/concentration, time to initial and repeat dosing, training/background of personnel administering drug), source of funding, and results relevant to each Key Question. We calculated relative risks and 95% confidence intervals if necessary for included outcomes, from data reported in the studies. All data abstractions were reviewed by a second investigator for accuracy and discrepancies were resolved through discussion and consensus. See Appendix D for data abstraction of included studies.

Risk of Bias Assessment of Individual Studies

We assessed risk of bias of included studies using predefined criteria. Two investigators independently assessed risk of bias. Disagreements were resolved by consensus. Our approach for assessing risk of bias is based on the Methods Guide for Effectiveness and Comparative Effectiveness Reviews.⁵² We adapted criteria for assessing risk of bias from the U.S. Preventive Services Task Force.⁶¹ For RCTs, risk of bias assessment criteria included randomization and allocation concealment methods, comparability of groups at baseline, blinding, attrition, use of intention-to-treat analysis, and prespecification of outcomes. For cohort studies, assessment criteria were based on patient selection methods; comparability of groups at baseline; methods used to ascertain exposures, confounders, and outcomes; blinding of outcomes assessors; attrition and missing data; and statistical analysis of potential confounders. For uncontrolled longitudinal studies, we used the same criteria as for cohort studies, but did not assess comparability of groups at baseline or statistical adjustment for confounders. Studies were rated as “low risk of bias,” “medium risk of bias,” or “high risk of bias” based on the presence and seriousness of methodological shortcomings; uncontrolled studies were rated high risk of bias since they can only address the comparative effectiveness questions addressed in this review indirectly.

Studies rated “low risk of bias” are considered to have the least risk of bias, and their results are generally considered valid. “Low risk of bias” studies include clear descriptions of the population, setting, interventions, and comparison groups; a valid method for allocation of patients to treatment; low dropout rates and clear reporting of dropouts; appropriate means for preventing bias; and appropriate measurement of outcomes.

Studies rated “medium risk of bias” are susceptible to some bias, though not necessarily enough to invalidate the results. These studies may not meet all the criteria for a rating of low risk of bias, but no flaw likely to cause major bias. The study may be missing information, making it difficult to assess limitations and potential problems. The “medium risk of bias” category is broad, and studies with this rating vary in their strengths and weaknesses. The results of some medium risk of bias studies are likely to be valid, while others may be only possibly valid.

Studies rated “high risk of bias” have significant flaws that imply biases of various types that may invalidate the results. They have a serious or “fatal” flaw in design, analysis, or reporting; large amounts of missing information; discrepancies in reporting; or serious problems in the delivery of the intervention. In general, cohort studies that do not perform adjustment for potential confounders were assessed as “high risk of bias.” The results of high risk of bias studies are at least as likely to reflect flaws in the study design as the true difference between the compared interventions. We did not exclude studies rated high risk of bias a priori, but high risk of bias studies were considered to be less reliable than low or medium risk of bias studies when synthesizing the evidence, such as when discrepancies between studies are present.

See Appendix E for the risk of bias assessments of included studies.

Assessing Research Applicability

Applicability is defined as the extent to which the effects observed in published studies are likely to reflect the expected results when a specific intervention is applied to the population of interest under “real-world” conditions.⁵² It is an indicator of the extent to which research included in a review might be useful for informing clinical decisions in specific situations. Because applicability depends on the perspective of the user of the review, we did not assign a rating for applicability (such as “high” or “low”). We used the PICOTS framework to consider the applicability of the evidence base for each Key Question. For example, we examined the characteristics of the patient populations (e.g., age [including the proportion of patients >65 years of age], duration of overdose, and opioids involved in the overdose, if known [in particular, involvement of long-acting opioids or high-potency synthetic opioids]); and study setting (e.g., field versus clinical setting). Variability in the studies may limit the ability to generalize results to other populations and settings.

Data Synthesis and Rating the Body of Evidence

We constructed evidence tables with study characteristics, results, and risk of bias ratings for all included studies, and summary tables to highlight the main findings. Given the small number of studies for each Key Question and clinical and methodological heterogeneity among the studies, we determined that meta-analysis was not indicated. Rather, we synthesized studies qualitatively. We graded the strength of evidence for each Key Question and comparison for prioritized clinical outcomes (mortality, time to reversal of symptoms, recurrence of overdose symptoms, respiratory or cardiac arrest, rates/severity of drug withdrawal, and combativeness) by using the approach described in the AHRQ Methods Guide for Comparative Effectiveness and Effectiveness Reviews.⁵² One investigator performed the initial strength of evidence assessment and discussed with the entire team to reach consensus. Assessments were based on the following domains:

• Study limitations, based on the overall risk of bias across studies (low, medium, or high)
• Consistency of results across studies (consistent, inconsistent, or unable to determine when only one study was available)
• Directness of the evidence linking the intervention and health outcomes (direct or indirect)
• Precision of the estimate of effect, based on the number and size of studies and confidence intervals for the estimates (precise or imprecise)
• Reporting bias, based on whether the studies defined and reported primary outcomes and whether we identified relevant unpublished studies (suspected or undetected)

Based on our assessments on the domains described above, we graded the strength of evidence for each Key Question using the four key categories recommended in the Methods Guide for Comparative Effectiveness and Effectiveness Reviews.⁵² Randomized controlled trials start as “high” strength of evidence and were graded down based on the presence of deficiencies in the domains. Observational studies start as low, and can be upgraded for factors such as presence of a dose-response relationship, large magnitude of effects, or plausible confounders increasing reported estimates. A “high” grade indicates high confidence that the evidence reflects the true effect and that further research is very unlikely to change our confidence in the estimate of effect. A “moderate” grade indicates moderate confidence that the evidence reflects the true effect and further research may change the estimate. A “low” grade indicates low confidence that the evidence reflects the true effect and further research is likely to change the confidence in the estimate of effect and could increase the confidence in the estimate. An “insufficient” grade indicates evidence either is unavailable or is too limited to permit any conclusion, due to extreme study limitations, inconsistency, imprecision, or reporting bias.

See Appendix F for the strength of evidence table.

Peer Review and Public Commentary

Peer reviewers and federal partners were invited to provide comments on the draft report. The AHRQ Task Order Officers and an EPC Associate Editor also provided comments and editorial review. The EPC considered comments prior to finalization of the report and completed a disposition of all peer review comments, which will be posted after publication of the final report on the AHRQ public Web site.