Search Strategy

Searches were executed between September 2013 and June 2014. We conducted search update during peer review of the draft report. We developed search strategies using a combination of subject headings (i.e., controlled vocabulary) and keywords (Appendix A).We included broad terms for psychosocial interventions, as well as interventions by name (e.g., “Parent-Child Interaction Therapy”, “Incredible Years”, and “Positive Parenting Program”). We included terms to describe drug classes and individual agents. We built the search strategies in tandem with the refinement of the KQs and Analytic Framework to ensure that the literature retrieval was representative of the project scope. The preliminary results were vetted by clinical and methodologic subject matter experts. We did not conduct a separate search for longitudinal cohort studies of adverse events, but did conduct a separate search for existing systematic reviews and requested drug package inserts to obtain information on harms.

Inclusion and Exclusion Criteria

The inclusion and exclusion criteria for the review were derived from our understanding of the literature, refinement of the review topic with the Task Order Officer and Key Informants, and feedback on the KQs obtained during the public posting period.

Population

The target population for this review is children under 18 years of age who are being treated for a disruptive behavior (Table 2). Eligible studies had to focus on the treatment of the disruptive behavior and include children exhibiting disruptive behaviors as a primary problem (e.g., conduct disorder, oppositional defiant disorder, and intermittent explosive disorder). We considered also, studies that included subjects who were not diagnosed with a disorder but who were being treated for disruptive behaviors that were measured by and found to be above the clinical cutoff on a validated measure.

Table 2

Case definition for disruptive behavior.

We included studies of interventions that targeted parents of children with a disruptive behavior if the study explicitly defined the eligible patient population to include a child with a disruptive behavior (as defined above) and the study reported one or more child outcome. We excluded studies of disruptive behavior secondary to other conditions (e.g., treatment of substance abuse, developmental delay, intellectual disability, pediatric bipolar disorder). In the case of ADHD, we excluded studies of ADHD-related disruptive behaviors but included studies of non-ADHD-related disruptive behaviors in populations of children with ADHD if the children were identified as also having another disruptive behavior disorder. Our quantitative analysis further excluded studies that did not report baseline and end of treatment means and standard deviations using one of the three most commonly used outcome measures.

Data Extraction

We created data extraction forms to collect detailed information on the study characteristics, interventions, comparators, outcomes, outcome measures, and study quality and/or risk of bias (see Study Characteristics and Outcomes Data Files in the Systematic Review Data Repository). We enumerated the variables most important to this topic with input from Key Informants and Technical Experts and used the extraction forms to record participant characteristics, intervention characteristics, outcomes, and potential modifiers of treatment effects from each included study. The forms included detailed instructions and labels to reinforce coding reliability and consisted of items with mutually exclusive and exhaustive answer options to promote consistency. A senior level team member reviewed the data extraction against the original articles for quality control. The study and data abstraction forms were used to develop summary tables across selected groups of studies.

We recorded descriptive data for each study that met the full text screening criteria including study design, year, location, setting, randomization, blinding, elements of study quality, and related publications. We flagged related publications and extracted nonduplicate study data. We categorized location by country with the exception of Puerto Rico, which we categorized separately from the U.S. due to cultural differences in the study population. We recorded the source of funding and authors' competing interest disclosures for all studies included in the review.

We recorded intervention characteristics and components in detail, noting data elements not reported or unavailable from the primary or related study publications. We classified interventions according to their treatment components, specifically: 1) interventions including only a child component; 2) interventions including only a parent component; and 3) multicomponent interventions. Multicomponent interventions were defined as those that included two or more of a child component, parent component, or other component (e.g., teacher component, family together component).

We categorized outcomes broadly as behavioral or functional. We extracted information on how the outcome was measured and the outcome measurement time points. We include broad measures of quality of life and social functioning.

To assess the evidence on harms, we first collected adverse outcomes reported in studies included for effectiveness. We also identified the evidence for harms of pharmacologic interventions used to treat disruptive behavior reported in the gray literature, including integrated safety reports from the U.S. Food and Drug Administration's regulatory documents.

We recorded potential modifiers to determine whether specific variables affected treatment response. We anticipated that patient age and certain disorder characteristics (such as disease severity) would be robust predictors of outcomes.

We also extracted information on intervention delivery, intervention setting, and environmental factors (e.g., parental engagement) that may account for variations in observed treatment effects. The potential modifiers represent categories of variables that we anticipated may be linked to treatment effects. We extracted the reported variables from included studies and organized the information into meaningful groups to permit syntheses.

Data Management

We registered the review protocol (Registration #CRD42014007552) with PROSPERO, an international database of prospectively registered systematic reviews in health and social care. We used DistillerSR (Evidence Partners, Ottawa, Canada) for screening references. We tracked the literature search retrieval and screening results in EndNote. We used forms to extract the study data, and transferred the data to Excel. We deposited the data that were used in the meta-analyses into the Systematic Review Data Repository (SRDR) system.

Determining Risk of Bias Ratings

We assigned studies an overall rating of “low,” “moderate,” or “high” risk of bias. We expected RCTs to receive positive assessments for questions about randomization, allocation concealment, and blinding in order to be designated “low risk of bias.” We considered the feasibility of blinding in psychosocial studies and did not downgrade where it would have been impossible. Cohort studies that received positive scores on all items were assessed as “low risk of bias.” Cohort studies with two or fewer negative ratings were assessed as “moderate risk of bias” and studies with more than two negative scores were assessed as “high risk of bias.” We required that studies assessed for harms reporting receive a positive rating (i.e., affirmative response) on all four questions to receive a rating of “good.” Studies with at least three positive responses were considered “fair” quality and those with less than three positive responses were assessed as “poor” quality.

Qualitative Synthesis of Results

We qualitatively synthesized the literature based on the data extracted (described above) for each Key Question. We present behavioral outcomes (KQ1 and KQ2) and harms data (KQ5) in summary tables within the text. For the qualitative summary of KQ1, we organized the results by age (preschool, school age, and teenage) and characterized the studies as those that evaluated a child-only, a parent only, or a multicomponent intervention, based on the active treatment arm. We defined multicomponent interventions as those that included two or more of a child component, parent component, or other component (e.g., teacher component, family together component). We further grouped the summary of studies for KQ1 by named interventions (e.g., PCIT, Triple P, and Incredible Years) where possible. This categorization provided an organizational structure to characterize the literature and highlight key findings for similar interventions. For KQ2 we grouped the studies by individual pharmacologic agent or by pharmacologic class.

Quantitative Synthesis of Results

We developed a Bayesian multivariate, mixed treatment (network) meta-analysis to address the comparative effectiveness of psychosocial interventions for improving behavioral outcomes for children treated for disruptive behaviors (Key Question 1). We used Bayesian multivariate, mixed treatment (network) meta-analytic methods^77-79 to use both direct and indirect evidence for comparing a large suite of treatments. Network meta-analysis allows for a broader, integrated view of the available evidence, allowing for the relative merits of a set of treatments to be more readily compared. This approach borrows strength from indirect comparisons of interventions that have not been compared head-to-head in the same study. By combining direct and indirect evidence in the same framework, the resulting meta-analysis may be more robust, with more precise meta-estimates, than traditional meta-analyses. In the absence of network meta-analysis, we would have been compelled to construct a number of smaller, separate meta-analyses that would have been less powerful and less comprehensive, with more evidence excluded relative to a unified network meta-analysis. Further, our model was multivariate, in the sense that multiple outcome measures were considered simultaneously; this improves the analysis by recognizing that outcomes are correlated, estimating that correlation directly as part of the analysis. We present additional details of the meta-analysis methods in Appendix D.

Twenty-eight of the 66 studies included in the qualitative review in KQ1 met the additional criteria for inclusion in our meta-analysis. These additional criteria were that the study was an RCT that reported baseline and end-of-treatment means and standard deviations using one (or more) of the three most prevalent of the 16 instruments used in this literature to examine parent reported outcomes: (1) Eyberg Child Behavior Inventory (ECBI), Intensity Subscale; (2) ECBI, Problem Subscale; and (3) Child Behavior Checklist (CBCL), Externalizing (T-score) (see Appendix E for a description of the instruments). Other instruments were not included in the analysis because of heterogeneity of constructs examined and an inadequate number of studies per measure.

To account for the large suite of interventions employed by the constituent studies, we classified the study arms of each included study according to their treatment components or as a control. Specifically, the treatment arms of each study were classified as one of the following types: (1) interventions including only a child component; (2) interventions including only a parent component; and (3) multicomponent interventions. Multicomponent interventions were defined as those that included two or more of a child component, parent component, or other component (e.g., teacher component, family together component). All interventions classified as multicomponent included a parent component. Study arms not identified by any of these three classes were defined as a control arm (i.e., waitlist control or treatment-as-usual arm). Recognizing that these treatment categories are broad, encompassing a range of specific interventions, each component was modeled as a random effect. This allowed for variation in treatment effect within each class, due to factors not explicitly modeled here. All measurement instruments shared the same study arm treatment effect in our model.

Studies were included in the meta-analysis if they reported baseline and end-of-treatment means and standard deviations from one of the three metrics listed above. The baseline was subtracted from the end-of-treatment mean and used as the response measure, along with the sum of their standard deviations. The three outcomes were modeled jointly as a multivariate normal likelihood, with any unmeasured outcomes treated as missing data; this allowed for the covariance among measures to be accounted for and estimated.

The age of subjects in each study arm was included in the model as a categorical covariate, broadly grouped into either prekindergarten, preteen child or teenage categories. The preteen child was used as the baseline value because it was the most prevalent among studies. The age covariate was combined additively with the intervention component effects and control/treatment-as-usual means to model the observed treatment differences relative to baseline. Though we considered age-by-treatment interactions, there was not enough balance among the age and treatment combinations to include them in the final model.

All unknown parameters were given weakly-informative prior distributions and estimated using Markov chain Monte Carlo⁸⁰ methods via the PyMC 2.3 software package.⁸¹ The model was run for 200,000 iterations, with the first 150,000 samples conservatively discarded as burn-in, leaving 50,000 for inference.

Incorporating Existing Systematic Reviews

We located reviews published between 2005 and 2014 and evaluated each for relevance using the review PICOTS (Appendix B). We summarize review data from relevant psychosocial and pharmacologic interventions in the “Discussion” section of the report and in a table in Appendix F. For the systematic reviews reporting harms, we assessed quality using AMSTAR⁷³ and summarized the findings in KQ5.

Strength of Evidence Assessments

We referenced the recommendations from the AHRQ EHC Methods Guidance and updated guidance for grading the strength of a body of evidence.^82,83 In accordance with the methods guidance, we first assessed and graded “domains” using established concepts of the quantity and quality of evidence, and coherence or consistency of findings. Two senior staff independently graded the body of evidence; disagreements were resolved through discussion.

We assessed strength of evidence for the direction or estimate of effect for the behavioral outcomes and interventions listed in Table 3.

Table 3

Selected outcomes and comparisons for the strength of evidence assessments.

We assessed an overall evidence grade based on the ratings for the following domains: study limitations; directness; consistency; precision; and reporting bias. We considered additional domains, as appropriate: dose-response association, plausible confounding, and strength of association (i.e., magnitude of effect). The fifth required domain, reporting bias, includes publication bias, selective outcome reporting, and selective analysis reporting.⁸² To assess publication bias in the pharmacologic literature, we sought study protocols and data from regulatory sources and compared this information to the results in the published literature. The issue of publication bias in psychological science is difficult to address given the current lack of standards regarding the registration of study protocols in social sciences. We attempted to minimize the potential for bias introduced by the “file drawer effect” (i.e., nonpublication of studies with nonsignificant results) by expanding the literature search to include unpublished sources (e.g., meeting abstracts) and asking Key Informants about current research or developments in the field that may not yet be published.

Overall Strength of Evidence

We summarize the four grades (high, moderate, low, and insufficient) we used for the overall assessment of the body of evidence in Table 4 (adapted from the AHRQ “Methods Updated Guidance for Grading the Strength of a Body of Evidence”⁸²). When no studies were available for an outcome or comparison of interest, we graded the evidence as insufficient.

Table 4

Strength of evidence grades and definitions.