Topic Refinement and Review Protocol

During the topic development and refinement processes, we engaged in a public process to develop a draft and final protocol for the review. We generated an analytic framework, preliminary Key Questions (KQs), and preliminary inclusion/exclusion criteria in the form of PICOTS (populations, interventions, comparators, outcomes, timing, settings). Information provided by the topic nominator helped guide our processes; similarly, other methods and content experts and KIs provided insights to help formulate our procedures. We also conducted a scan of the relevant literature. We carried out preliminary literature searches and discussions with KIs to develop appropriate KQs.

We worked with five KIs during the topic refinement; all five also served as members of our Technical Expert Panel (TEP) for this report. They represented critical care medicine, pulmonology, infectious disease, infectious disease pharmacy, and payers. TEP members participated in conference calls and discussions through email at several points: review the analytic framework, KQs, and PICOTS; discuss the preliminary assessment of the literature; provide input on the information and categories included in evidence tables; and comment on the data analysis plan.

Our KQs were posted for public comment on AHRQ's EHC program Web site (www.effectivehealthcare.ahrq.gov) from March 22, 2013, through April 18, 2013. We revised them as needed after reviewing the comments and discussing them with the TEP; specifically, we decided to include dose-monitoring studies, in which no therapeutic drug monitoring occurs during the studies but which apply PK/PD principles. We then drafted a protocol for the review that was posted on the same Web site. Its PROSPERO registration number is CRD42013005309.

Literature Search Strategy

Inclusion and Exclusion Criteria

We developed eligibility (inclusion and exclusion) criteria with respect to PICOTS and study designs and durations for each KQ (Table 2). We required that studies measure and report at least one of our specified outcomes. For both intermediate outcomes and health outcomes, randomized controlled trials (RCTs), nonrandomized controlled trials, and prospective cohort studies were eligible. For adverse effects data, case-control and retrospective cohort studies were also eligible.

Table 2

Eligibility criteria for review of PK/PD measures for hospital-acquired pneumonia (HAP).

We required studies to have a comparator to be included. Because of this requirement, studies lacking a comparator PK/PD target goal were not eligible; similarly, retrospective cohort studies without an appropriate comparator group were not eligible. Our goal was not to examine an individual drug's performance; rather, we focused on use of PK/PD measures to guide and optimize treatment.

Thus, many vancomycin dosing studies for S. aureus pneumonia using PK/PD measures would not be eligible if they did not prospectively compare two or more different dosing approaches, such as targeting two different troughs. Studies that retrospectively examined the peak or trough values that different patients achieved and related those data to the MIC of the organism would also not be included. Furthermore, studies evaluating extended interval aminoglycoside dosing were not included if they did not have a prospective comparator group.

We excluded studies of fungal pneumonia in this review because fungal infections would involve a different set of PICOTS from those found in the literature for bacterial lung infections. Because the report scope was limited to HAP, ventilator-acquired pneumonia (VAP), or health-care-acquired pneumonia (HCAP), we also did not include studies of community-acquired pneumonia or other pneumonias in which treatment began in a setting other than the hospital. Ventilator-associated tracheobronchitis (VAT) that has not become VAP would not meet the inclusion criteria for our review. PK/PD studies of VAT alone would need to be separate from VAP because the concentration of drug at the site of infection differs.

In addition, because of the report's focus on pneumonia, we did not include studies of shock, sepsis, or other infections that did not provide data for HAP patients. As stated in the introduction, the lung is a unique organ for drug penetration. Thus, serum concentrations for other conditions, such as sepsis, do not necessarily correlate with optimizing dosing for pneumonia.

Finally, we excluded studies in which serum concentration had been measured without comparing different serum concentration targets, because this type of intervention would be considered standard of care. For that reason, these practices do not constitute a study design for examining optimization of PK/PD measures to inform treatment decisions.

Risk of Bias Assessment of Individual Studies

To assess the risk of bias (i.e., internal validity) of studies, we applied predefined criteria based on the AHRQ Methods Guide.⁵¹ This approach uses questions to assess selection bias, confounding, performance bias, detection bias, and attrition bias—that is, it addresses issues of adequacy of randomization, allocation concealment, similarity of groups at baseline, masking, attrition, whether intention-to-treat analysis was used, method of handling dropouts and missing data, validity and reliability of outcome measures, and treatment fidelity.

Two independent reviewers assessed risk of bias for each study. Disagreements between the two reviewers were resolved by discussion and consensus or by consulting a third member of the team.

Studies are rated as low, medium, or high risk of bias. In general terms, results from a study assessed as having low risk of bias are considered to be valid. A study with medium risk of bias is susceptible to some risk of bias but probably not enough to invalidate its results. A study assessed as high risk of bias has significant risk of bias (e.g., stemming from serious issues in design, conduct, or analysis) that may invalidate its results.

Data Synthesis

We did not find multiple studies for any comparison of interest that reported similar outcomes; for that reason, we could not consider quantitative synthesis (i.e., meta-analysis) of the data from the included studies. All analyses in this review are, therefore, qualitative. We synthesized data from the included studies in tabular and narrative format. Synthesized evidence was organized by KQ.

Strength of Evidence of the Body of Evidence

We graded the strength of evidence based on the guidance established for the EPC program.⁵² Developed to grade the overall strength of a body of evidence, this approach incorporates four required domains: risk of bias (including study design and aggregate quality), consistency, directness, and precision of the evidence. Reviewers can also consider other optional domains that may be relevant for some scenarios; these include dose-response association, plausible confounding that would decrease the observed effect, strength of association (i.e., magnitude of effect), and publication bias.

Table 3 defines the grades of evidence that we assigned. We graded the strength of the body of evidence for major outcomes and comparisons relating to the three KQs stated above. Two reviewers assessed each domain for each key outcome and resolved differences by consensus. For each assessment, one of the two reviewers was always an experienced, senior investigator. The overall grade was based on a qualitative decision taking into account the ratings for the four required domains, and, if relevant, ratings of the other domains.

Table 3

Definition of the grades of overall strength of evidence.

We graded the strength of evidence for the outcomes deemed to be of greatest importance to clinicians and other stakeholders. Tables showing our assessments for each domain and the resulting strength of evidence grades for each KQ, organized by intervention-comparison pair and outcome, appear in the results section.

Applicability

We assessed the applicability of individual studies as well as the applicability of the body of evidence following guidance from the AHRQ Methods Guide.⁵³ For individual studies, we examined factors that may limit applicability based on the PICOTS framework. Such factors may be associated with heterogeneity of treatment effect or the ability to generalize the effectiveness of an intervention to use in everyday practice. Some factors identified a priori that could limit the applicability of evidence for this review included the following: severity of illness, whether studies enrolled patients with chronic lung diseases, and settings.

Peer Review and Public Commentary

The AHRQ Task Order Officer (TOO) and an AHRQ associate editor (a senior member of another EPC) reviewed the draft report before peer review and public comment. The draft report (revised as needed) was sent to invited peer reviewers and simultaneously uploaded to the AHRQ Web site where it was available for public comment for 28 days.

We collated all reviewer comments (both invited and from the public) and addressed them individually. We documented all our responses to these comments in a disposition of comments document, which will be posted on the AHRQ EHC program Web site about 3 months after Web publication of the evidence report. The authors of the report have final discretion as to how the report will be revised based on the reviewer comments, with oversight by the TOO and associate editor.