Background:

For childhood attention-deficit/hyperactivity disorder (ADHD), treatments that combine behavioral and pharmacologic interventions have proven to be more efficacious than either behavioral treatment alone or medication alone. Despite the strong evidence base for these 2 treatment types, both remain vastly underused for a host of reasons. Further, combined treatment has not been systematically tested among adolescents. The issue is further compounded by a lack of understanding within the field about comorbidity issues related to substance use (SU) among adolescents with ADHD.

Objectives:

The primary objective of this cluster randomized trial was to compare the effectiveness of behavioral-only vs integrated (behavioral plus medication decision-making) interventions for adolescents with ADHD in outpatient services on participant functioning and service use. A secondary objective was to examine patient-centered outcomes with respect to heterogeneity of effects among adolescents with and without SU problems and individual differences in effects on participant functioning based on ADHD medication use.

Methods:

A cluster randomized intention-to-treat (ITT) longitudinal design was used to test the effects of behavioral-only vs integrated (behavioral plus medication decision-making) treatment in routine care for all adolescents who met diagnostic criteria for ADHD in 5 outpatient behavioral health clinics. In the behavioral-only condition, therapists were trained in an academic support intervention protocol for adolescents with ADHD focused on homework management and organizational training. In the integrated condition, in addition to academic support interventions, therapists were trained in a family-based medication decision-making protocol. Clinicians were randomly assigned to study condition in a 2:1 ratio favoring the integrated condition. Therapist fidelity to each treatment protocol was assessed via both therapist-report checklists and observation of session recordings. Primary outcomes were ADHD symptoms, comorbid problems, executive functioning, and school functioning. Secondary outcomes were treatment attendance, medication acceptance, and treatment satisfaction. Primary outcomes were analyzed using latent growth curve modeling to examine the impact of treatment condition on change over time in client outcomes. Secondary outcomes were tested via multistep heterogeneity of effects analyses focused on moderator effects.

Results:

The study enrolled 145 clients (n = 53 behavioral only, n = 92 integrated) treated by 49 therapists (n = 20 in behavioral only, n = 29 in integrated). Adolescent participants were 72% male; 42% White non-Hispanic, 37% Hispanic American, 15% African American, and 6% more than 1 race; and average age was 14.8 years. Therapist participants (N = 49) included 82% women; 63% White non-Hispanic, 29% Hispanic American, and 10% some other race/ethnicity; average postgraduate experience was 5.1 years. All enrolled clients and therapists were included in study analyses (ITT). Therapist self-reports of protocol adherence indicated strong treatment fidelity overall: Observer ratings indicated overall fidelity with regard to delivery of Changing Academic Support in the Home for Adolescents with ADHD (the behavioral protocol for boosting academic outcomes) but not with regard to medication integration protocol fidelity. Analysis of primary outcomes found that for most client outcomes, there were no meaningful differences between treatment conditions. Regarding ADHD symptoms, integrated care produced a greater decline in adolescent-reported ADHD inattentive symptoms. However, there were no differences for adolescent-reported hyperactive symptoms, caregiver-reported inattentive symptoms, or caregiver-reported hyperactive symptoms. Regarding comorbid problems, integrated care produced a greater decline in number of delinquent acts compared with behavioral-only care However, there were no between-group differences for adolescent-report internalizing symptoms, adolescent-report externalizing symptoms, caregiver-report internalizing symptoms, caregiver-report externalizing symptoms, or SU. No between-group differences were found for executive functioning (self-regulation, self-organization), school functioning (school grades, academic self-efficacy, homework problems, minutes spent doing homework), or medication use. Post hoc analyses identified some between-group differences between adolescents who did, vs did not, use substances at baseline. No intervention harms or other adverse events were reported.

Conclusions:

Significant additive effects for medication decision-making interventions were found for a minority of the main effects tested and for a minority of the subgroups of adolescents examined in post hoc analyses, across clinical, developmental, and medication outcomes.

Limitations:

The number of participating sites was too small to test for site effects. Participating therapists at each site were self-selected, creating some limitations to the generalizability of findings. The 2:1 randomization ratio was not purely achieved due to initially randomized therapists being excluded because they were never assigned a study case (n = 33), as well as the need to assign a small subset of therapists directly to the condition to maintain critical mass for group consultation. In the absence of a no-treatment control group, the magnitude of research assessment effects on client outcome cannot be estimated, although it is presumed to be equally distributed across conditions. Because multiple hypotheses were tested, there is an increased possibility that 1 or more reported analyses represents a chance finding.

Aim 1: Comparative Impact on Client Functioning

Aim 2: Comparative Impact on Service Use

Aim 3: Examine Patient-Centered Outcomes

Project Stakeholder Launch Meetings

First, to better understand the needs and viewpoints of families of adolescents with ADHD, the principal investigator (PI) met with 14 caregivers of adolescents with ADHD to discuss the various conceptual and strategic dimensions of patient-centered, family-based decision-making about ADHD medications for adolescents. This meeting was hosted in November 2014 by Children and Adults with Attention-Deficit/Hyperactivity Disorder, a national nonprofit that provides education, advocacy, and support for individuals with ADHD. This group voiced strong concerns that ADHD-related symptoms remained an impediment to optimal school performance for their children; that families needed much more information about the potential benefits, harms, and adverse effects of ADHD medications that pertain specifically to the teenage years; that they harbored significant reservations about the impact that stimulant medications might exert on SU vulnerability in their children, along with strong potential for misuse and diversion of prescribed medications; and that they welcomed guidance from professionals (pediatricians, school counselors, therapists) but did not feel that they could routinely access information about ADHD medication from sources other than the internet.

Second, before study materials were finalized, Drs Aaron Hogue and Steven Evans led a stakeholder meeting with 12 outpatient clinicians in February 2015, during which the clinical protocols were vetted. Feedback from this meeting provided expert user-centered insight into the design of study materials and training protocols. This group expressed a strong need for (1) standardized protocols that could be readily implemented in routine behavioral care but were flexible enough so that content could be tailored to meet the needs of a broad range of developmental and clinical profiles among adolescents and families; (2) user-friendly psychoeducation materials about ADHD medications for adolescents that covered the vast array of medical and behavioral concerns voiced by caregivers and teens, including concerns about providing medication to youth who use substances and/or have numerous SU risk factors, and that are easily understood and delivered by nonmedical providers; and (3) pragmatic behavioral interventions for boosting academic study habits of youth with ADHD, as well as corresponding guidance for caregivers on how to support their children effectively in this area, which is typically a source of substantial family conflict and parental stress.

Third, in March 2016, a larger executive stakeholder meeting was convened at the National Center on Addiction and Substance Abuse (now Center on Addiction) to present preliminary findings from year 1 of the project and to solicit feedback on the successes and challenges encountered during the first year. This meeting included a specific advisory group of 7 representatives from the research community, including the Child Mind Institute, ADD Resource Center, and Columbia University Medical Center; 8 representatives from one of our partnering treatment sites, ADHD educators, and national advocates; 4 representatives from youth-focused community agencies; and 3 youth and their families who had previously received services for ADHD that included early pilot versions of the 2 study protocols. This meeting provided an effective forum for discussing the need for psychoeducation for families of adolescents with ADHD, adolescent perceptions of medication use, and educators' roles in addressing ADHD in the classroom.

Semiannual Researcher Meetings

These meetings were attended by the PI, co-investigator (Dr Evans), and 2 senior clinical researchers who were independent of the study. The meetings reviewed the ongoing progress of site clinical training, family enrollment and home-based assessment, qualitative data collection, and quantitative data collection. These meetings yielded detailed procedures for improving study recruitment (eg, deploying research staff on site once per week at each partnering site to troubleshoot client identification and gain permission for initial research contact) and finalizing the design of qualitative data collection from study therapists.

Triannual Clinical Stakeholder Meetings

Every 4 months (triannually), 2 members of our research staff hosted a voluntary stakeholder meeting at each of the 5 enrolled treatment sites for therapists and supervisors to provide feedback on intervention and study procedures. These meetings were attended by about 6 to 8 therapists and 1 supervisor in each sitting at each site. Meeting goals included discussing therapists' use and implementation of the protocol, suggestions for protocol improvements, and methods to increase therapist and supervisor engagement in the study. Following stakeholder meetings, the research staff convened to discuss approaches to implement the stakeholders' feedback.

In the first triannual stakeholder meeting, supervisors from Outreach, one of our first treatment sites, reflected on their experiences with recruiting study participants, therapist training, and therapist use of the protocols. An essential piece of feedback the supervisors provided was that it would be useful to see summaries of the therapy procedures self-report form that the study therapists were submitting. In response, the research staff created monthly site-specific data reports and discussed them with therapists and supervisors during site fidelity meetings.

Before the August 2016 triannual stakeholder meeting, therapists had received feedback from research staff regarding compiled data from the therapist self-report checklist of treatment procedures used in sessions with study clients. After coding therapists' audio-recorded sessions and reviewing therapists' checklists, the research staff detected a pattern of therapists underrating their use of the CASH-AA module 1 intervention—specifically the motivational, reframing, and relabeling intervention—in the checklists. In response, during the August 2016 stakeholder meeting, therapists suggested that it would be helpful to receive more in-depth training in the module 1 intervention. As a result, the research team revamped the existing training materials for new sites and also conducted 1-hour CASH-AA module 1 trainings at already enrolled sites to help meet this therapist-identified need. Further, therapist feedback inspired the research staff to offer one-on-one protocol trainings upon request.

Other therapist-inspired additions included the creation of a weekly email bulletin and 5 one-page protocol review sheets. These were developed in response to therapist feedback that it was difficult to remember the protocols and cumbersome to quickly review the manuals while also gathering enough information to prepare and deliver the interventions successfully. As such, the weekly bulletins were created to provide brief (≤500 words) reviews of the different elements of the interventions. During subsequent triannual stakeholder meetings, therapists suggested that the bulletins had become a way to keep them updated on current ADHD research and other aspects of study progress also reported in the bulletins (eg, what interviews were like for the families, how many interviews had been completed, what rates of ADHD symptoms were reported at baseline by parents and teens). In addition, research staff created 5 one-page protocol review sheets: 2 briefly describing the CASH-AA and MIP interventions; 1 CASH-AA specific page describing how to adapt the protocol for teens who were no longer in school; 2 additional CASH-AA pages describing how to use the protocols for teens experiencing anxiety/depression; and 1 page for addressing family stress. The last 2 were developed as a direct response to feedback from therapists who were concerned that focusing solely on ADHD is not the most effective treatment strategy for teens struggling with various comorbidities. Taken together, these 5 one-pagers act as a quick reference guide to the protocols and offer ways to adapt the protocols to diverse adolescent mental health needs.

As the study continued, therapists expressed interest in receiving feedback on their session recordings and hearing about other clinicians' (eg, colleagues, supervisors, clinicians at other sites) implementations of the protocols with various types of clients and family structures, as well as across a variety of treatment settings (eg, family therapy clinics, SU clinics). Every month, a research staff member selected a 10-minute excerpt of a previously coded session recording that exemplified high-fidelity protocol delivery, and played the excerpt in monthly fidelity monitoring meetings. After a recording was played, research staff led a guided discussion addressing the strengths of the recorded session and alternative use of the protocols. As such, study therapists were able to reflect critically on both their own protocol implementation practices and those of their peers. This information was then incorporated into ongoing clinical consultation at each site.

Study Overview

The primary objective of this cluster randomized trial was to compare the effectiveness of behavioral-only vs integrated (behavioral plus medication decision-making) interventions for adolescents with ADHD in outpatient behavioral services. The behavioral intervention, CASH-AA, contains 3 components: family-based motivational interventions pertaining to ADHD and academic performance, academic skills training, and family-school partnership. The medication decision-making intervention, MIP, contains 3 components: education about ADHD and ADHD medication, family-based medication decision-making, and integrated medication management. The study aimed to (1) compare the effects of 2 evidence-based treatment options for adolescents with ADHD on behavior symptoms, quality of life (QOL), and service use (aims 1 and 2); and (2) generate new evidence on a patient-centered treatment provision that aligns with family-specific treatment goals (aim 3). We considered testing the 2 protocols in an efficacy trial, in which we could use narrow inclusion criteria to select a relatively homogeneous profile of cases and also selectively hire, train, and supervise research therapists delivering care in a controlled setting. In consultation with PCORI staff, we decided that testing the protocols in an effectiveness trial provided the greatest generalizability and overall benefit. We also carefully considered a research design in which integrated care (CASH-AA plus MIP) would be tested against a treatment-as-usual condition. Again in consultation with PCORI, we decided that testing integrated care (CASH-AA plus MIP) vs behavioral-only care (CASH-AA only) provided the most rigorous test of the unique effects of the family-based medication decision-making protocol (ie, MIP), which was deemed a top scientific priority for the discipline of patient-centered care.

Study Setting

All 5 partnering treatment sites were outpatient behavioral health clinics in the New York City metro area. There were 2 community-based mental health clinics, 1 hospital-based mental health clinic within a department of psychiatry, 1 community-based ASU clinic, and 1 community-based clinic co-licensed in mental health and SU services. All full-time therapists and clinical trainees at each site who treated adolescent clients and who volunteered to participate were accepted into the study; based on site administrator report, between 75% and 85% of therapists at each site volunteered, and this percentage did not vary significantly across sites. Therapists at each site managed comparable client caseloads (including individual, family, and group sessions), and each site routinely prescribed weekly treatment sessions and offered in-house psychiatric support. Therapists at each site routinely received a comparable amount of weekly individual and/or group in-house supervision (60-90 minutes per person).

Participants: Therapists

The 2 protocols were delivered in existing services by 49 agency clinicians. Randomization occurred at the therapist level: Volunteer therapists at each site were randomly assigned to a study condition. Thus, “therapist” was the clustering variable in the design, and study clients were nested within study therapists. To maximize delivery of the MIP across study cases, we weighted randomization to favor 2:1 proportional assignment to the integrated condition. Note that it was not possible to randomize also at the client level (ie, doubly randomized design²⁴) because partner clinics were not able to assign clients randomly to study therapists without disrupting routine client flow and therapist caseload management. All full-time clinicians and clinical trainees at each of the 5 partner sites were invited to participate in the study on a volunteer basis. All therapists at each site were consented, provided data on clinical training background and site organizational characteristics, and then convened for a 90-minute on-site introductory training in the CASH-AA protocol. At the end of the training, therapists were randomly assigned to either the behavioral-only condition (CASH-AA only) or the integrated condition (CASH-AA plus MIP). Those randomly assigned to the integrated condition were then invited to participate in an additional 90-minute on-site introductory training in the MIP. These same training and randomization procedures were followed whenever new therapists were hired at each site. In a few instances, when the number of active therapists in a study condition at a given site dipped to <3 due to staff turnover, the next therapist to volunteer at that site was assigned directly to the low-member condition to maintain critical mass for group-based protocol consultation meetings. Of the total 49 therapists enrolled in the study across all sites and assigned a case, <10% were assigned group membership in this fashion, rather than being randomized; this proportion did not significantly differ among study sites. As shown in the CONSORT diagram (Figure 1), 82 therapists consented to participate and were assigned to a study condition (50 integrated, 32 behavioral only); of these, 49 (60%) eventually received a study case and therefore became study therapists (29 integrated, 20 behavioral only).

Figure 1

CONSORT Participant Referral and Flow Diagram.

Participants: Clients

All study clients were recruited from existing clinical referral streams at the 5 partner sites. Recruitment followed a 2-gate strategy: (1) Intake staff at each site pitched the study to all families during the clinic's routine intake process; and (2) study therapists at partner sites pitched the study to families of those adolescents on their existing caseloads who had not already been recruited for the study. At both gates, the clinic staff member attempted to obtain permission to share family contact information with research staff. When permission was granted, clinicians contacted research staff by phone to relay contact information.

Research staff attempted to contact the primary caregiver of referred families by phone within 24 hours of referral receipt. During the initial call, research staff described the main features of study participation; collected data on adolescent age, race/ethnicity, and sex; and obtained verbal consent to administer a brief ADHD screening tool.²⁵ Families who screened positive (ie, caregiver score on the screening tool adding to at least 10 on inattention/disorganization and/or hyperactivity/impulsivity scales rating each of 10 symptoms on a Likert-type scale from 0 to 3 for the adolescent) were immediately recruited to schedule an in-person home-based baseline assessment to determine full study eligibility; those who refused to schedule a baseline interview were asked about their primary reason(s) for refusing. During the baseline assessment, caregivers and adolescents were consented and interviewed separately; caregivers consented for themselves and their teenagers, and teenagers assented for themselves. Caregiver assessments were administered in the preferred language: 81% English and 19% Spanish. Assessment measures consisted of a structured clinical interview and audio computer-assisted self-report measures of adolescent clinical symptoms, school performance, other aspects of developmental functioning, family characteristics, history of treatment engagement and medication use, and legal system involvement. Each family member received an honorarium in gift cards for completing the baseline assessment, which typically lasted 60 to 90 minutes.

Study inclusion criteria were (1) adolescent was aged 12 to 18 years; (2) primary caregiver was able to participate in treatment; (3) adolescent met DSM-5 diagnostic criteria for ADHD, based on either caregiver or adolescent report; (4) adolescent was not enrolled in any behavioral counseling at any other site; (5) caregiver expressed desire, and adolescent expressed willingness, to participate in treatment at the partner site; and (6) family had health insurance benefits that were accepted for routine behavioral services by the given partner sites (all sites accepted a broad range of insurance plans, including Medicaid). Exclusion criteria were mental retardation or autism spectrum disorder; psychiatric or other medical illness requiring hospitalization; current psychotic symptoms; active suicidal ideation; or severe SU problems that require immediate relief (detox or residential placement). No inclusion or exclusion criteria were related to medication status; that is, youth were permitted to be either taking or not taking medications of any kind at study enrollment, and to change their status at any time during the study.

Interventions and Comparators

Therapists in the behavioral-only condition were trained in the use of CASH-AA; therapists in the integrated condition were trained in the use of CASH-AA and MIP. Full descriptions of the protocols are contained in Appendix A. Note that the study placed no conditions or restrictions on the natural tenure of each case in treatment (ie, each study case remained enrolled in treatment at the full discretion of the partnering clinics, and protocols were implemented at full discretion of the study therapists). No matter the length in treatment for any given case, for every case (1) research staff conducted home-based assessment interviews at 3, 6, and 12 months after the baseline assessment interview; and (2) study therapists were asked to submit protocol fidelity data (post-session self-report checklists, session audio recordings) for all sessions convened throughout the course of the 1-year research follow-up period.

Fidelity Evaluation

We used a 2-pronged approach to evaluate therapists' fidelity to CASH-AA and MIP: (1) therapist reports of the extent to which each of their sessions with study participants focused on ADHD, ADHD medication, and elements of CASH-AA and MIP; and (2) observational coding of audio recordings of therapy sessions conducted by research staff who were blind to study condition and had participated in extensive and ongoing training in observation techniques and tools by the PI and project director. Study therapists were asked to record as many sessions as possible. It was deemed important to collect therapist self-report fidelity data in addition to session audiotapes for 2 reasons: (1) We believed we would collect a substantially greater number of self-report checklists covering a much larger proportion of convened sessions, thereby providing a more extensive scope on treatment fidelity; and (2) we anticipated that observational data might confirm that some self-report fidelity data points were reliable, enabling interpretation of those data for the much larger proportion of sessions for which there was therapist-report fidelity.

Audio recordings were collected for more than half of the sessions for which therapists submitted session data (54.7% [n = 571]). Study therapists were encouraged to record as many sessions as possible for all families who consented to allow session recordings; therapists were informed which families declined to allow recordings at the consent process, which amounted to 14% of all participants. Of the 49 therapists enrolled in the study (n = 20 in behavioral only; n = 29 in integrated), 14 (29% of all therapists: 4 [17%] behavioral only, 10 [35%] integrated) submitted at least 1 audio-recorded session. It is not known to what degree the recorded sessions are representative of all convened sessions. The main reasons voiced by study therapists for not recording treatment sessions were therapist belief that recording sessions would negatively impact treatment; therapist discomfort at being recorded; therapist belief that session recording and data submission were overly burdensome; therapist report that participants declined to allow recording during sessions; and therapist failure to access recording equipment before the session.

Of those sessions that were recorded, a random selection (68.0% [n = 388 sessions]) was observed and coded by at least 1 trained research team member. Of these, 242 were double-coded for CASH-AA use, and 58 were double-coded for ADHD, ADHD medication, and MIP use. Interclass correlation coefficients (ICCs) were computed to determine the interrater reliability of coder pairs that was documented for these double-coded tapes. Results indicated that coders were fairly to highly reliable in their coding of the number of minutes in session spent focused on ADHD, ADHD medication, CASH-AA protocol use, and MIP use (ICCs, 0.69-0.86). Therefore, the remainder of coded tapes were single-coded.

According to therapist report, of the total number of sessions for which data on protocol delivery were submitted (N = 1051), 354 sessions (34% of the total) contained some delivery of the CASH-AA protocol, and 109 sessions (10% of the total) contained some delivery of the MIP. According to observational coder report, of the total number of sessions for which a submitted audio recording was coded (n = 501 and 387 for CASH-AA and MIP, respectively), 336 sessions (67% of the total coded) contained some delivery of the CASH-AA protocol, and 117 sessions (30% of the total coded) contained some delivery of the MIP.

Interrater reliability coefficients were again computed, this time examining the reliability between observational fidelity evaluations (the mean of the 2 coder ratings was used for those that were double-coded) and therapist-reported fidelity evaluations, to determine the accuracy of therapist self-reports of their time in session focused on ADHD, ADHD medication, CASH-AA use, and MIP use. Interrater reliability was low for ADHD and CASH-AA and MIP use (ICCs, 0.10-0.26) but was acceptable for session time focused on ADHD medication (ICC, 0.53). Therapist-reported and observational mean fidelity scores are shown in Table 1. Independent sample t tests were conducted to compare conditions on fidelity scores. Because these were preliminary descriptive analyses, therapist clustering was not accounted for.

Table 1

Session Time Focused on ADHD, ADHD Medication, and Study Protocol Use in Minutes.

Therapist reports of fidelity indicated that those in the integrated condition spent significantly more time focused on ADHD generally than those in the behavioral-only condition (t = 3.15; P < .01); however, this was not supported by coder reports of fidelity (t = 0.37; P = .72). Therapist reports of time spent in session focused on ADHD medication identified a trend-level difference, with therapists in the integrated condition reporting more time spent discussing ADHD medication than therapists in the behavioral-only condition (t = 1.69; P = .09). Therapists across both conditions reported no significant difference in amounts of time spent on elements of the CASH-AA protocol in session (t = 0.49; P = .63), which was supported by coder reports (t = 0.21; P = .84). In contrast, therapists in the integrated condition reported spending significantly more session time focused on elements of the MIP (t = 2.53; P < .05). This was not supported by coder observations, however, in which coders identified no significant difference in use of elements of the MIP between the 2 conditions (t = 1.16; P = .25).

Several primary themes emerged from these analyses. First, therapist self-reports of protocol adherence indicated strong treatment fidelity overall: (1) There were no differences between conditions in delivery of CASH-AA protocol elements, which was common to both study groups; (2) integrated therapists delivered a greater amount of MIP elements; (3) integrated therapists devoted more session time to discussing ADHD issues (significant effect) and ADHD medication (trend-level effect). Second, coder ratings based on the content of the audio-recorded sessions were congruent with therapist ratings only for amount of time spent discussing ADHD medication; they were discrepant for all other fidelity ratings. Third, coder ratings indicated overall fidelity with regard to CASH-AA delivery (ie, they reported no significant difference between conditions) but not with regard to MIP fidelity (ie, they reported no significant differences among conditions on MIP delivery, discussion of ADHD issues, or discussion of ADHD medication).

Study Outcomes: Primary (Aim 1) Adolescent Functioning

ADHD symptoms were assessed using the Mini International Neuropsychiatric Interview (MINI), v.5.0.²⁶ The MINI is a brief structured diagnostic interview that assesses DSM-IV diagnoses in adolescent and adult populations. The MINI is specifically designed to be administered by lay interviewers and has demonstrated solid interrater and test-retest reliability in 2 international samples of psychiatric and nonpsychiatric patients,²⁷ as well as excellent convergent validity with both the Structured Clinical Interview for DSM Disorders (SCID) and the Composite International Diagnostic Interview (CIDI).^26-28 Two adolescent-reported and 2 caregiver-reported dimensional variables were calculated: total number of symptoms endorsed (of 9) on the Inattentive/Disorganized subscale and total (of 9) on the Hyperactive/Impulsive subscale.

Externalizing and internalizing symptoms were measured with the Child Behavior Checklist (caregiver report)²⁹ and Youth Self-Report³⁰ (adolescent report), which are parallel measures of youth behavior problems supported by extensive evidence encompassing reliability, validity, and clinical utility³¹ and used with a wide range of adolescent samples.^32-34 Each contains a summary scale of externalizing symptoms (rule-breaking, oppositional, and aggressive behaviors; 32 and 35 items, respectively) and internalizing symptoms (anxious/depressed, withdrawn, and somatic complaint behaviors; 31 and 32 items, respectively). Higher scores correspond to more symptoms; scores on each item range from 0 to 2, and the study variable was calculated by summing items within each scale.

Delinquency was assessed using the National Youth Survey Self-Report Delinquency Scale (SRD),³⁵ a self-report delinquency scale used in the National Youth Survey³⁶ to assess adolescent criminal behavior and peer criminal behavior according to 5 subscales: Total Delinquency, General Theft, Crimes Against Persons, Index Offenses, and Drug Sales. The SRD is a well-validated instrument³⁷ used extensively with African American and Hispanic populations³⁸ and with adolescent clinical samples.^1,22 Adolescents reported on the number of times they engaged in various overt and covert delinquent acts.

SU was captured with the Comprehensive Addiction Severity Index for Adolescents (CASI-A),^39,40 a semistructured clinical interview that yields information about SU during the past 30 days as well as the severity of SU risk factors and consequences. The CASI-A has demonstrated strong reliability and validity with clinical record reviews of adolescents receiving inpatient psychiatric or substance abuse treatment³⁹ and with other diagnostic interviews such as the CIDI and DiSC (behavior assessment based on 4 personality traits: dominance, influence, steadiness, and conscientiousness).⁴⁰ Teens reported on the number of days they used any alcohol or illegal drugs in each month of follow-up. An adolescent was categorized as “SU+” if they (1) met DSM-5 criteria for SU disorder at baseline or (2) were enrolled in treatment at the SU treatment site; otherwise, they were categorized as “SU−”. Of the 145 participants in the analyzed group (Figure 1), 46 (32%) were SU+ and the remainder SU−.

Executive functioning was measured with the Behavior Rating Inventory of Executive Function (BRIEF), a caregiver-report measure of behavioral problems that are linked to executive functioning and commonly observed in youth with ADHD.^41,42 Gioia et al⁴¹ reported good convergent and discriminant validity between the BRIEF and similar behavioral rating systems as well as test-retest reliability statistics ranging from 0.79 to 0.88 during a 2-week period; internal consistency ranging from α = .80 to .98; and interrater reliability between parent and teacher responses of r = 0.32. The BRIEF has been validated in ADHD outpatient samples⁴³ and teens with mixed clinical diagnoses.⁴⁴ This study used 2 scales, the Behavioral Regulation Index global scale (Inhibition, Behavioral Shift, and Emotional Control subscales) and the Plan/Organize scale. Higher scores correspond to greater difficulty with behavior regulation and organization; scores on each item range from 0 to 2, and the study variable was calculated by summing the scores on each scale.

School functioning was measured with 4 variables. School grades and minutes spent doing homework were measured using the CASI-A³⁹ (described earlier), a semistructured adolescent clinical interview with strong reliability and validity^39,40 and concurrent validity for treatment-seeking adolescents.²⁵ This measure captured school grades (responses mapped grade numerals onto a continuous scale ranging from 0 to 7), with higher scores indicating better grades. Adolescents also reported on the number of minutes spent doing homework each day. Academic self-efficacy was measured using 4 items from the Self-Efficacy for Learning and Performance subscale of the Motivated Strategies for Learning Questionnaire.⁴⁵ Adolescents indicated to what extent they felt certain they could understand the most difficult material presented in texts and could master the skills being taught, and felt confident they could understand the most complex material presented by their teachers and could do an excellent job on assignments and tests. Higher scores indicated higher academic self-efficacy; scores on each item range from 0 to 3, and the study variable was calculated by averaging the z-score of the 4 items. Homework management was measured with the inattention/avoidance items of the Homework Problems Checklist.^46,47 This caregiver-reported measure consists of 11 items that probe a number of homework problem behaviors (eg, complaining about homework, waiting until the last minute, producing messy homework) over the past 2 weeks. Higher scores indicate more problems with homework; scores on each item range from 0 to 3, and the study variable was calculated by summing the scale items.

Study Outcomes: Secondary (Aim 2) Service Use

Treatment attendance (total number of individual, family, and group sessions attended) and medication management sessions were collected from agency records. Medication use, coded as “on” or “off” medication at each follow-up point, was captured with the Services Assessment for Children and Adolescents,⁴⁸ a parent-report structured interview that assesses attendance in outpatient services and use of psychiatric medications and has strong validity, test-retest reliability, and reliability between parent and child reports.^48,49 Note that the study ultimately did not collect data on treatment satisfaction. Just before enrolling initial study cases, we completed review of treatment satisfaction data from a randomized trial conducted with a highly similar client population in the same geographic area.⁵⁰ Those data clearly indicated that virtually every study participant (adolescents and caregivers) reported satisfaction at or near the top score of the given measure, effectively eliminating meaningful variance due to a pronounced ceiling effect. For this reason, and to reduce interview burden on clients in the current study, we elected not to collect treatment satisfaction data in this trial.

Sample Size Calculations and Power

Sample size for all 3 study aims was N = 145. Power analyses for main effects and interactions under nested designs were conducted using established formulas^51,52 and replicated in 2 software packages.^53,54 While accounting for the ICC, which typically reduces the effective sample size compared with the size realized under the assumption of independence between clients in a cluster, we noted a range of power to detect the specified interaction effect (of approximately 4% of explained variance). We assumed holding type I error to 5%, that 140 cases would be nested within 15 to 20 study therapists, and that the ICC of case outcomes within therapist would fall within the typical range of 0.005 to 0.03,^55,56 although recent experiences suggest ICCs toward the upper range. Following established formulas for nested designs,^57,58 Table 2 illustrates study power to detect effects of R² given our assumptions. At a conservatively estimated magnitude of an ICC of 0.03, we calculated that we would be able to detect medium to large main effects when testing for client outcomes (inattention and hyperactivity symptoms), and at least moderate power to test for interactions between study condition and baseline SU (SU+ vs SU−) whenever there were less-than-large effects.

Table 2

Study Power to Detect Effects.

Time Frame for the Study

The study began in April 2015 and concluded in August 2018. Between April and June 2015, project staff were hired and trained, and materials were finalized. Between June and August 2015, clinicians, supervisors, and administrative staff at the study sites were introduced to the study, consented to participation, and participated in trainings on the 2 study protocols. In August 2015, 2 families participated in pilot screening and baseline interviews to assist the research team in finalizing the data collection processes and materials. The first participating families were recruited, screened, and enrolled in September 2015, and enrollment continued until February 2018. Data collection was completed with families and study sites on August 31, 2018.

Data Collection and Sources

Data were collected from families participating in the study and from clinicians and clinical site records. At the completion of the baseline interview (described earlier), study-eligible families were immediately offered participation in follow-up home-based interviews at 3, 6, and 12 months after baseline; families who accepted were then immediately consented and scheduled for follow-up interviews. Client participants and research staff assessing client outcomes were blind to treatment condition. All initial screens for eligibility were conducted by telephone with the primary caregiver. All family baseline interviews and most follow-up interviews were conducted in the family home with both the caregiver and adolescent present with the research staff member who was conducting the interviews. On occasion, families with scheduling conflicts were offered the opportunity to participate in telephone-based follow-up interviews; 11 families chose this option for at least 1 follow-up interview. In 6 cases, adolescents were transferred to residential treatment or detained by the juvenile justice system, which meant they were unable to participate in follow-up interviews in person. In these cases, caregivers were interviewed in the family home, and for adolescents in residential treatment, efforts were made with the treatment site supervisor to schedule supervised phone-based interviews. Retention rates for each of the follow-up interviews were high (94%-96%), and only 2 families neglected to participate in all follow-up interviews.

Session data were collected from therapists via audio recording and from questionnaires completed after each session with study families. Research staff members traveled to each study site once every 2 weeks to collect audio recordings and questionnaires. Between July and August 2018, research staff worked with site administrative staff to collect data pertaining to the number of sessions attended by each study family. Length of client participation in treatment services varied based on routine clinical decision-making by the partner agency.

Analytical and Statistical Approaches

All data were housed at Center on Addiction, and all analyses were conducted by Center on Addiction research staff on computers belonging to the center. Staff at partnering sites did not have access to raw study data.

Changes to the Original Study Protocol

Two changes were made to the original study protocol; both were approved by PCORI and the IRB office at Center on Addiction. First, an amendment was made to add a fifth partnering treatment site about halfway through the study. The addition of this fifth site, one that focused on treating ASU problems, ensured that we would reach our goal of a final sample consisting of a large subpopulation of adolescents with ADHD and co-occurring SU. In addition, this fifth treatment site allowed us to add 12 additional therapists to our sample.

Second, an amendment was made to incorporate a qualitative data component into the study protocol in the form of focus groups with participating clinicians and supervisors at partnering treatment sites. Focus group questions asked participants to reflect on the study protocols and materials provided, in particular, whether and how therapists found the CASH-AA and MIP interventions useful, which specific elements and modules fit most seamlessly into the treatment they typically offer, which adolescents seem to benefit the most from these protocols, how successful the protocols had been for adolescents, and how they might suggest adapting the protocols to better suit the population and treatment setting. Focus groups were conducted at each of the 5 partnering treatment sites during the final year of the study and yielded rich information to improve the substance and design of the protocols in ways that were meaningful to study stakeholders. The procedures and findings of qualitative data analyses are described in Appendix B.

Sample Characteristics

Demographics and other clinical characteristics of the client sample (N = 145), as well as baseline values of all client outcome variables, are presented in Table 3a for the whole sample and separately by condition. As shown in Table 3a, the adolescents enrolled in the study were 72% male and 42% White non-Hispanic, 37% Hispanic American, 15% African American, and 6% more than 1 race; the average age was 14.8 years. There were no significant differences between study conditions in any of the demographic and clinical characteristics listed in Table 3a, indicating that randomization at the therapist level resulted in equivalent distribution between treatment conditions on all measured characteristics.

Table 3a

Baseline Demographic and Clinical Characteristics for the Full Sample and Separately by Condition.

Therapist participants (N = 49) were 82% female and 63% White non-Hispanic, 29% Hispanic American, and 10% some other race/ethnicity; their average amount of postgraduate experience was 5.1 years. A total of 72% had a terminal master's-level degree, 10% had a bachelor's-level degree, 8% were at the PhD level, 6% were at the MD level, and 4% were unknown. There were no condition differences in therapist characteristics (Table 3b).

Table 3b

Therapist Characteristics by Condition.

Descriptive Statistics on Outcome Variables

The CONSORT diagram (Figure 1) depicts the flow of clients into the study and the interview completion rates. Of the 305 families referred to the study, 48 (15.7%) were determined to be ineligible by research staff before participating in the initial phone screen: 32 families refused to participate in the phone screen, 8 were unable to be reached by phone, 3 had dropped out of treatment by the time they were reached by phone, 2 had children who were too young to be eligible, 2 were dropped due to the family dealing with other family crises, and 1 had been referred to inpatient treatment by the time they were reached by phone.

Descriptive statistics for all outcome variables at each time point are presented in Table 4 separately for the behavioral-only and integrated conditions. The distribution for delinquency showed significant departure from normality, hence the use of 2-part LGC models (described previously).

Table 4

Descriptive Statistics for All Outcome Variables by Treatment Condition.

Testing Shape of Change Across Conditions

Chi-square difference tests of nested models were used to determine whether linear or quadratic change best characterized each outcome. Quadratic models produced the best fit to the data for the following outcome variables: adolescent-reported inattentive/disorganized symptoms (henceforth “inattentive symptoms” or “inattention”) and hyperactive/impulsive symptoms (henceforth “hyperactive symptoms” or “hyperactivity”); caregiver-reported hyperactive symptoms; and adolescent-reported internalizing and externalizing symptoms. For all other outcomes, linear models produced the best fit. Both linear and quadratic parameter estimates are presented for quadratic models, and linear parameter estimates only are presented for linear models.

We first tested unconditional models for all outcomes to obtain the unadjusted growth factors (Table 5). Change in the outcome is indicated by a significant linear or quadratic effect parameter. As described earlier in the analysis plan, when the quadratic effect was statistically significant (ie, P < .05), we did not interpret the linear effect. Significant quadratic effects were found for adolescent-reported inattentive and hyperactivity symptoms, as well as caregiver-reported hyperactivity symptoms, indicating an uptick in symptoms following the initial decline. A similar pattern of significant quadratic effects was found for adolescent-reported internalizing and externalizing symptoms, with initial declines followed by some increases in symptoms. Linear declines were found for caregiver-reported inattentive symptoms and caregiver-reported internalizing and externalizing symptoms. There was no change in delinquency or drug use for the overall sample, for which there was lesser prevalence and severity of symptoms than for ADHD. Linear improvements in self-regulation and self-organization were found, as well as in homework problems. There was no overall change in school grades, academic self-efficacy, or minutes spent doing homework.

Table 5

Unconditional Models Testing Change Across Condition for All Outcomes.

Comparative Impact on Client Functioning (Aim 1)

To test for condition effects on reductions in symptoms (hypothesis 1) and improvements in QOL (ie, developmental functioning; hypothesis 2), we tested condition effects for each outcome, controlling for nuisance covariates (Table 6). A condition effect was found on the quadratic effect for adolescent-reported ADHD inattentive symptoms. As seen in Table 6, the data show no condition impacts on the intercept (baseline value) or linear effect for ADHD inattentive symptoms. The pseudo-z test for condition effects on the quadratic effect parameter is statistically significant, suggesting a group difference in the shape of change over time. Figure 2 shows that behavioral-only clients showed an initially steeper rate of decline in ADHD symptoms, followed by a later increase in symptoms after 6 months, whereas integrated clients showed a steadier rate of decline in ADHD inattentive symptoms over the follow-up period. No other condition effects on ADHD symptoms were found. Regarding comorbid problems, an effect of condition was found for number of delinquent acts. Among clients who engaged in any delinquency, integrated clients showed greater declines in the number of delinquent acts compared with clients in the behavioral-only condition. No other condition effects on comorbid problems or QOL were found. Overall, statistically significant differences favoring the integrated condition were found for 3 of the 18 outcomes analyzed.

Table 6

Condition Effects on Outcomes Controlling for Baseline Covariates (Sex, Age, Race, Baseline SU).

Figure 2

Condition Effects on Adolescent-Reported ADHD Inattentive Symptoms.

Comparative Impact on Service Use (Aim 2)

We conducted linear regression controlling for nuisance covariates to test for condition differences in behavioral therapy attendance (sum of individual, family, and group sessions) and logistic regression controlling for nuisance covariates to test for condition differences in whether clients ever attended a medication management session (hypothesis 3). Due to a low number of medication management sessions (ie, sessions in which family members were seen by a physician to discuss medication) attended (41% of the sample never attended any), this variable was dichotomized for analysis. No condition effects were found for either behavioral therapy or medication management attendance (Table 7). To examine heterogeneity of effects as in aim 3, we also tested for potential moderating effects of sex, age, race, SU identification, and whether the adolescent started ADHD medication during follow-up. There was a moderating effect of baseline SU on behavioral therapy attendance. Probing this interaction revealed a condition effect on behavioral therapy attendance for teens not using substances (B [SE] = 6.36 [2.87]; 95% CI, 0.62-12.10; β = 2.21; P = .027; n = 88), indicating that among teens not using substances, clients in the integrated condition attended more behavioral sessions on average (mean [SD], 19.4 [14.3] sessions) than did behavioral-only clients (mean [SD], 12.8 [11.4] sessions).

Table 7

Condition Effects on Service Use Outcomes Controlling for Baseline Covariates (Sex, Age, Race, and Baseline SU).

To test hypothesis 4, we examined condition differences in ADHD medication uptake. Across conditions, of the 84 adolescents who were not taking ADHD medication at baseline, 24 (29%) initiated medication at some point during the follow-up period. Of the 24 adolescents who initiated medication, 8 were in the behavioral-only condition (33%), and 16 were in the integrated condition (67%). This distribution of adolescents initiating medication did not differ across conditions; that is, 29% in each condition initiated ADHD medication during follow-up. We also examined condition differences in adolescent use of ADHD medication at each time point using the same LGC modeling used for aim 1. No condition effect on ADHD medication use was found (see last rows of Table 6).

HTE (Aim 3)

Results of analyses of HTE by baseline SU (research question 1) are presented in Table 8. No effects were found on executive functioning or ADHD medication use. An interaction between baseline SU and condition was found for caregiver-reported hyperactive symptoms. Regarding comorbid problems, interactions between baseline SU and condition were found for 3 outcomes: adolescent-reported externalizing symptoms, caregiver-reported externalizing symptoms, and delinquent acts. All interactions were probed by testing effects of condition on the outcome variable separately for adolescents identified as not using substances at baseline vs those identified as using substances at baseline. The following condition effects were found for teens not using substances: There was a condition effect on the quadratic effect for adolescent-reported externalizing (B [SE] = 1.15 [0.39]; 95% CI, 0.37-1.93; pseudo-z = 2.91; P = .004) (Figure 3), indicating greater initial declines in symptoms for clients in the integrated condition that leveled off over time. There were no condition effects on caregiver-reported externalizing or delinquency for teens not using substances. The following condition effects were found for teens using substances: There was a condition effect on caregiver-reported hyperactivity symptoms (B [SE] = 0.52 [0.46]; 95% CI, −0.40 to 1.44; pseudo-z = 2.92; P = .004), suggesting that among substance users, behavioral-only clients showed more improvement in hyperactivity symptoms than did integrated clients. There was a condition effect on the linear term for adolescent-reported externalizing (B [SE] = 2.12 [0.66]; 95% CI, 0.80-3.44; pseudo-z = 3.20; P = .001), indicating increases in symptoms for clients in the integrated condition.

Table 8

Models Testing Main Effects of Condition and Baseline SU and Moderating Effect of Baseline SU on Outcomes.

Figure 3

Condition Effects on Change in Adolescent-Reported Externalizing Symptoms for Clients Who Were Not Using Substances at Baseline.

There was a condition effect on the linear effect for caregiver-reported externalizing (B [SE] = 2.17 [0.84]; 95% CI, 0.49-3.85; pseudo-z = 2.59; P = .010; n = 46), indicating greater declines in symptoms among substance-using clients in the behavioral-only condition. Although there were no condition effects on delinquency in either group, the parameter estimates were in opposite directions, which likely accounted for the interaction effect.

Results of analyses testing HTE by sex (research question 1) are presented in Table 9. Sex × condition interactions were found for multiple outcomes; we report them here to illustrate the overall pattern of results favoring the integrated condition. Interaction effects were detected for adolescent-reported inattentive symptoms and adolescent-reported hyperactivity symptoms. Probing the interactions revealed condition effects for boys only. There were condition effects on the quadratic effect for inattention (B [SE] = −0.45 [0.13]; 95% CI, −0.71 to −0.19; pseudo-z = −3.57; P = .001; n = 104), suggesting an initial decline followed by an increase in symptoms for the behavioral-only condition and a more steady decline in symptoms for the integrated condition. Probing the interaction for hyperactivity symptoms revealed no condition effects in either group. There was an interaction for adolescent-reported internalizing symptoms. Probing the interaction revealed a significant effect on the linear slope for girls (B [SE] = −2.00 (0.94); 95% CI, −3.88 to −0.12; pseudo-z = −2.13; P = .033), indicating greater declines in symptoms among girls in the integrated condition. There were no other effects on comorbid problems or ADHD medication use. Regarding executive functioning, there was a sex × condition interaction on self-regulation. Probing the interaction revealed a condition effect for girls (B [SE] = −1.71 [0.84]; 95% CI, −3.39 to −0.03; pseudo-z = −2.03; P = .042; n = 41), suggesting greater improvement in self-regulation among girls in the integrated condition. There was also a main effect of sex on self-organization, with girls showing greater improvement in self-organization over the follow-up period than boys. Regarding school functioning, a sex × condition interaction was found for minutes spent doing homework. Probing the interaction revealed a condition effect for girls only (B [SE] = 15.58 [5.73]; 95% CI, 4.12-27.04; pseudo-z = 2.72; P = .007; n = 41), indicating greater increases in time spent doing homework among girls in the integrated condition. There was also a main effect of sex on homework problems, with girls showing greater declines in homework problems over the follow-up period.

Table 9

Models Testing Main Effects of Condition and Sex and Moderating Effects of Sex on Outcomes.

Results of analyses testing HTE by age (research question 1) are presented in Table 10. Main effects of age were found on caregiver-reported inattention and caregiver-reported hyperactivity (Figure 4). Across conditions, older teens showed greater declines in inattention and initial greater declines in hyperactivity, with symptoms increasing at the end of the follow-up period, whereas younger teens' symptoms continued to decline steadily. There was a significant age × condition interaction on the quadratic effect for adolescent-reported internalizing symptoms. Probing the interaction revealed no significant condition effects in either age group; the effects were in opposite directions. No other effects were found on comorbid problems or executive functioning. Regarding school functioning, an age × condition interaction was found for academic self-efficacy. For younger teens, clients in the integrated condition showed greater declines in academic self-efficacy (B [SE] = −0.14 [0.06]; 95% CI, −0.26 to −0.02; pseudo-z = −2.24; P = .025; n = 72), and for older teens, clients in the integrated condition showed greater increases in academic self-efficacy (B [SE] = 0.14 [0.05]; 95% CI, 0.04-0.24; pseudo-z = 2.56; P = .004; n = 73). Finally, there was an age × condition interaction for ADHD medication use. Probing this interaction revealed a condition effect for older teens (B [SE] = 1.19 [0.49]; 95% CI, 0.21-2.17; pseudo-z = 2.43; n = 73). Among older teens, clients in the integrated condition were more likely to use ADHD medication across the follow-up period compared with those in the behavioral-only condition.

Table 10

Models Testing Main Effects of Condition and Age and Moderating Effects of Age on Outcomes.

Figure 4

Age Differences in Caregiver-Reported Hyperactivity Symptoms.

Results of analyses testing HTE by race (research question 1) are presented in Table 11. As previously described, the moderator variable “race” was operationalized as a dichotomy: non-Hispanic White vs all others (ie, any other race). There were main effects of race on caregiver-reported inattention and self-organization. Non-Hispanic White adolescents showed greater declines in inattention and improvements in self-organization behaviors. Race × condition interactions were found for adolescent-reported externalizing and caregiver-reported externalizing. For adolescent externalizing, condition effects were found for non-White teens only. There was a condition effect on the quadratic effect for adolescent-reported externalizing among non-White teens (B [SE] = 0.84 [0.41]; 95% CI, 0.02-1.66; pseudo-z = 2.07; P = .038; n = 84). Clients in the integrated condition showed greater initial declines in externalizing, but these symptoms began to increase later in the follow-up period. For caregiver-reported externalizing, there was an effect of condition only for non-Hispanic White adolescents (B [SE] = 1.36 [0.53]; 95% CI, 0.30-2.42; pseudo-z = 2.56; P = .011; n = 61), with greater declines in symptoms seen for the behavioral-only condition.

Table 11

Models Testing Main Effects of Condition and Race and Moderating Effects of Race on Outcomes.

Analysis of Individual Differences (Research Question 2)

We examined ADHD medication status at follow-up as a potential moderator of condition effects on outcomes. If no moderating effects were found, we tested main effects of medication status on outcomes. In these analyses, we compared the 24 clients who were not taking medication at baseline but started medication at some point during the follow-up period with the 121 other clients. Note that we also ran these analyses limited to the subgroup of 84 who were not taking medication at baseline (24 started medication during follow-up vs 60 who did not), and several of the models did not converge due to the reduced sample size. Therefore, we present the full sample results in Table 12. A medication × condition interaction was found for caregiver-reported hyperactivity. Probing the interaction revealed a condition effect on the quadratic effect for those who started medication (B [SE] = 0.72 [0.25]; 95% CI, 0.22-1.22; pseudo-z = 2.92; P = .003; n = 24). Among those who started medication, integrated clients showed initial decreases in hyperactivity, but symptoms began to increase after 6 months. Behavioral-only clients showed the opposite pattern, with initial increases in symptoms followed by declines after 6 months. Regarding comorbid problems, there was a main effect of starting ADHD medication on the number of delinquent acts for those who engaged in any delinquency, with those who started medication showing greater increases in delinquency. There was a trend-level medication × condition interaction for adolescent-reported internalizing symptoms. Probing the interaction revealed no condition effects in either group; however, effects were in different directions, which likely accounted for the interaction term. Finally, there was a medication × condition interaction for SU. For those who started medication, there was a condition effect indicating that clients in the integrated condition were more likely to use substances across the follow-up period (B [SE] = 1.12 [0.44]; 95% CI, 0.24-2.0; pseudo-z = 2.57; P = .010; n = 24) than those in the behavioral-only condition.

Table 12

Models Testing Main Effects of Condition and ADHD Medication and Moderating Effects of ADHD Medication on Outcomes.

Section 1: Study Findings

Section 2: Main Study Contributions

Section 3: Strengths and Limitations

A main strength of the study was high external validity that affirms the generalizability of findings to real-world practice: Community therapists delivered the treatment protocols in everyday settings, and adolescents with ADHD from diverse backgrounds were enrolled in usual care, with psychiatric comorbidity the norm. Study therapists within each site were randomly assigned to study condition, which strengthened internal validity and muted therapist and site effects that might otherwise confound study results. Both adolescent and caregiver reports were used to gauge clinical progress, and ITT analyses were used for all main outcomes. Assessment of organizational culture and climate indicated that study sites were broadly representative of the national behavioral health treatment system.

Numerous limitations warrant mention. There was no treatment-as-usual comparison group, which limits conclusions that can be drawn about differences between the 2 intervention conditions (ie, CASH-AA plus MIP, and CASH-AA only) and usual practice. The study occurred in the New York City metropolitan area, thus imposing accompanying geographic limits to generalizability. The number of participating sites was too small to control for site clustering effects via random-effects analyses. The study was not adequately powered to support post hoc inspection of individual site effects using similarly rigorous analyses; thus, it is not known whether any of the partnering treatment sites performed at a significantly higher or lower level than others. Participating therapists at each site were self-selected, and although at most sites those volunteering to participate constituted most available clinical staff, they may not have been representative of all staff at a given site. The 2:1 randomization ratio was not, strictly speaking, achieved due to initially randomized therapists being excluded from the study because they were never assigned an eligible study case (n = 33), as well as the need to assign a small subset of therapists nonrandomly to study condition to maintain critical mass for group-based protocol consultation meetings.

Also, in the absence of a no-treatment control group, the degree to which the research interview process itself, independently of treatment effects, impacted assessment of client outcomes cannot be estimated, although such impacts are presumed equally distributed across study conditions. There were also key assessment limitations. Indices of client outcomes were collected from adolescents and caregivers only, to the exclusion of outcome data collected from school and clinical professionals who provide important and respectively unique perspectives on the behavioral functioning of youth with ADHD. Although pharmacologic studies often assess medication use via documentation of prescriptions filled and/or monthly pill counts by physicians (vs patient self-report), these more rigorous assessment methods were beyond the resources of this study.

Another limitation pertains to the conflicting reports of treatment fidelity submitted by study therapists (using post-session self-reports) vs observational coders (who reviewed session audio recordings). On the one hand, both sources concurred that there was no between-condition difference in delivery of CASH-AA protocol elements. This finding was as predicted and indicates experimental fidelity, given that both conditions were expected to implement the CASH-AA protocol in equal measure. Also indicative of experimental fidelity, therapists in the integrated condition (CASH-AA plus MIP) reported delivering a greater amount of MIP elements than those assigned to the CASH-AA intervention alone, and devoting more session time to discussing ADHD issues and ADHD medication, than did their counterparts in the behavioral-only (CASH-AA only) condition. However, coders did not report strong adherence to the MIP in the integrated condition; rather, they observed no significant differences between conditions in MIP delivery, discussion of ADHD issues, or discussion of ADHD medication. To be sure, it is typical to find lack of concordance between therapist self-report and observational coding of delivery of model-specific treatment techniques for all kinds of behavioral interventions. Studies attempting to show concordance between therapist self-ratings and observer ratings of fidelity have mostly produced disappointing results, casting doubt on whether therapists can reliably rate their own performance.⁸⁰ This includes research-trained clinicians delivering manualized treatment protocols^81,82 as well as frontline clinicians delivering native interventions in routine care.^32,83

The absence of observational coder support for expected MIP adherence evidence calls into question the MIP “dosage strength” received by participants enrolled in the integrated condition. It is possible that having a more structured (less flexible) protocol format, in which aspects of the MIP were prescribed for delivery in specified sessions and for specified amounts of session time, would have resulted in greater fidelity to the model and greater differentiation between conditions. It is also not known whether increasing protocol structure in this manner would have exerted a positive, negative, or neutral influence on therapist motivation to implement the study protocols. Finally, it is difficult to determine, under the conditions of this effectiveness trial in which both study protocols were flexibly implemented, whether each condition should be respectively characterized as having received a strong, adequate, or weak dose of each protocol. Additional research on dose-response effects, in which variation in the extensiveness of protocol implementation is correlated with variation in client outcomes, is needed to shed further empirical light on this issue.

Finally, an important caveat pertains to the nomenclature of the integrated condition. This condition was named “integrated” due to its emphasis on increasing the integration of pharmacologic interventions into behavioral care for adolescents with ADHD. However, the MIP is, itself, a behavioral intervention. Thus, whereas MIP is intended to facilitate greater integration of medication and behavioral treatments, the experimental design did not manipulate or otherwise assign ADHD medication status to any participant, nor did study procedures introduce inclusion criteria or service demands related to medication status of any kind. From this perspective, the “integrated” condition might be more accurately described as the “intent to integrate” condition. For certain, it should not be regarded as a bona fide “combined” treatment group, which refers to treatment packages whereby pharmacologic and behavioral interventions are paired by design,⁸⁴ such that every participant in the condition is expected to receive medication.

Appendix A.

Treatment Protocol Descriptions (PDF, 175K)

Appendix B.

Therapist Focus Groups (PDF, 242K)