Abstract

Background

Children and youth in foster care receive antipsychotic (AP) and other psychotropic medications at much higher rates than do other children.^1-5 These treatments have profound adverse health effects on the lives of vulnerable and often-traumatized children requiring oversight and monitoring. APs, mood stabilizers, stimulants, and antidepressants are among the main classes of medications widely received by children in foster care, each involving distinct adverse effect and risk profiles that require careful assessment before prescribing and careful monitoring once initiated. In some states, >20% of children in foster care have in earlier years received AP medication,⁶ and decisions about systems to oversee prescription of medications and management of risks (here, psychotropic oversight systems) raise difficult and controversial challenges for patients and their families, clinicians, child welfare/Medicaid systems, and other stakeholders. Oversight is critical given the following factors: the significant metabolic and other adverse effects of AP medications; the reviews of prescribing practices for Medicaid-enrolled children that suggest concerns about overprescribing, including a review of cases by the Health and Human Services (HHS) Inspector General; and the heightened in loco parentis responsibilities of a state government to ensure that treatments for children in foster care are appropriate.⁷

Given these concerns,⁸ states face difficult decisions in implementing oversight systems for publicly insured children, especially for children in foster care for whom the state, as guardian, has unique responsibility. Federal legislation and regulatory mandates require states to implement psychotropic oversight systems, which they do in highly varied ways. Oversight systems are complex and often poorly documented, and little evidence is available on the comparative effectiveness of alternative oversight system innovations and their impact on health outcomes. Although most states work with stakeholders to implement monitoring, it is unknown to what extent the outcomes are consistent with the values and priorities of those who are affected by them. These systems affect the welfare of hundreds of thousands of vulnerable children nationally, and they have implications for the large population of non–foster care, Medicaid-enrolled children, as well. Thus, it is vital to understand the choices available to state policy makers and the consequences of these choices.

To address these critical issues, we built on existing public/academic partnerships to examine the impact and effectiveness of alternative state psychotropic oversight systems, incorporating the perspectives of foster care alumni, caregivers, prescribers, and child welfare staff and using patient-centered measures of effectiveness prioritized as important by alumni, caregivers, and other stakeholders. A multidisciplinary team incorporating young adults with lived experience in the children's systems of mental health care examined the impact of differing, innovative oversight systems in Texas, Wisconsin, Washington, and Ohio. We identified, documented, and analyzed oversight systems in the study states (aim 1); elicited perspectives on and experiences of psychotropic medication treatments with essential stakeholders, including foster care alumni, caregivers, caseworkers, and clinicians, and their prioritization of effectiveness measures important to patients and stakeholders (aim 2); and examined stakeholder-prioritized outcomes of policies using Medicaid administrative data (aim 3). The study design used pre-post comparisons of treatment before and after state natural experiments in oversight policies and programs, with a sequential, stakeholder-engaged approach to identifying interventions most suitable and relevant for such analysis. As proposed, because the state initiatives and their implementation and timelines were not well documented, and because the selection of outcome measures was designed to reflect systematic elicitation of outcomes important to alumni and stakeholders and by stakeholder advisory board (SAB) input, the study was designed in a sequential, mixed-methods fashion in which aim 1 work informed the selection of specific state changes most appropriate and evaluable with pre-post analyses as “natural experiments” for aim 3 analysis. State interventions were selected for pre-post analysis in aim 3 based on criteria that included their significance as important types of initiatives likely to be considered by other states; implementation at a clearly defined, specific time point; the ability to clearly characterize the intervention as implemented; the availability of appropriate comparison groups; and stakeholder input. These decisions required information from aim 1 work, given the limited amount of documentation available on the programs as actually implemented. Work from both aim 1 and aim 2 informed the selection of measures in those analyses that were prioritized by stakeholders and considered most relevant to the nature and aims of the specific interventions studied, taking into account as well considerations of potential confounding by any concurrent changes identified as likely to confound pre-post comparisons. Considerations of potential confounding were particularly salient for candidate measures of potential unintended adverse effects on outcomes such as hospitalization and emergency department (ED) use; these were identified as likely to be subject to unmanageable confounding due to concurrent health system changes and were also given the lowest priority in the prioritization survey.

Participation of Patients and Other Stakeholders

The team at the Rutgers Center for Health Services Research (CHSR) has routinely engaged a range of stakeholder partners to inform research projects. This public/academic study was no exception; engagement of patients, families, and other stakeholders relevant to the comparative effectiveness research question of how best to promote safe psychotropic medication practices for children in foster care occurred before and throughout the research period across multiple domains and included collaborations related to research planning, problem solving, the conduct of research, and the dissemination of project findings. Young adults, alumni of the child welfare system, and their families are vulnerable and hard-to-reach populations requiring broad, varied, and creative outreach activities. Our success was advanced through important reciprocal partnerships with advocacy organization representatives, including Youth MOVE National, Foster Care Alumni of America (FCAA), and the National Foster Parent Association (NFPA). These groups provided conduits to the vulnerable population of interest and their families, and individuals from Youth MOVE National also served on the aim 2 research team. Patient engagement was advanced on multiple fronts through the SAB, individual and group discussions with federal and state policy leaders, consultation with state Medicaid data analysts, and clinical and subject matter experts. Engagement strategies throughout the project were broad-based and customized to address the various needs of stakeholders. Activities included conference calls; web-based briefings (8 briefings held during the project period); individual, in-person meetings (via state site visits, conferences, and face-to-face planning events); and regular advisory discussion. We met with Washington State policy leaders for a state-specific discussion of our findings and held 8 state-specific meetings relating to the interim results and descriptive trend analyses. Our work was carefully aligned with outcomes of interest that were prioritized by multiple stakeholder groups (see Best-Worst Scaling Experiment⁹), captured in the National Committee for Quality Assurance (NCQA) Healthcare Effectiveness Data and Information Set (HEDIS) AP measurement suite,¹⁰ vetted by the Centers for Medicare & Medicaid Services (CMS), and used as performance measures across Medicaid managed care organizations (MMCOs) nationally.

Engagement activities began with the development and design of the study to ensure stakeholder relevance and feasibility; the SAB first convened in 2012. This was particularly important to the aim 2 activities, which required customized approaches to recruitment and data collection, taking into account different state policy and program environments. The sample sizes and the heterogeneity achieved would not have been realized without stakeholder support. The most notable impact of stakeholder engagement on the study quality was the close reciprocal relationships built between the aim 2 research team and the advocate organizations. Building reciprocal, trusting relationships, consistent with authentic partnership principles and strategic co-learning activities that sought to build research and health policy capacity within the advocacy framework, was essential to the success of the project. The CHSR relationship with the leadership team of Youth MOVE National, other advocacy organizations, and select state partners is a salient example of how the research team included patients, families, and other stakeholders in the research enterprise. Our work with Youth MOVE National and states has been ongoing since 2013 when the CHSR team invited the organizations to become members of the aim 2 research team. Their activities were supported through grant resources comparable to contracts issued to other subject matter experts. Collaborative tasks included (1) study design and development of research questions, (2) data collection strategies and development and testing of instruments, (3) participant recruitment and screening, (4) patient-centered policy analysis, (5) thematic analysis, and (6) dissemination planning and identification of strategies for uptake and implementation. We sought to ensure the feasibility of the aim 2 protocol and at the same time build capacity with Youth MOVE National and policy leaders to participate with evaluation efforts locally and nationally. Our work with Youth MOVE National and other SAB members was initially time-intensive, meeting as a group quarterly and individually several times a month to customize recruitment efforts and work individually with states to ensure data submission. Throughout the project, we worked extensively with the SAB to develop, test, and refine data collection instruments and outcome metrics; build feasible recruitment strategies; collect data across a broad array of stakeholder groups, both nationally and in specific states; and plan and participate in dissemination activities, including co-presenting at national conferences, coauthoring research briefs, and participating in state-specific in-person/web-based discussion of project findings.

The perspectives of SAB members informed all aspects of the conduct of this research. Obtaining these perspectives required a revised research timeline and other adjustments so that we could partner with people who have lived experience within the state child welfare and children's mental health systems. A flexible stance to engagement activities and a robust investment in advocacy organizations were essential to our success. For example, this approach was critically important to the development and testing of the survey for stakeholder-prioritized outcomes, which we developed to identify those values and considerations that matter most to individuals who have lived experience with the child welfare and mental health systems of care. Based on input from the SAB, we expanded data collection to include the key informants in aim 1 to better understand the dissonance between how policy makers prioritize system outcomes and how other stakeholders prioritize system outcomes. Engaging in this way with patients, families, clinicians, and policy makers ensured a more patient-centered study.

Methods

Research Methods

Aim 1 Key Policy Informants

For data derived from interviews, we used framework analysis, a deliberate and systematic 5-step approach, to code and analyze qualitative interview data.^11,12 Framework analysis is well suited to research that aims to meet specific information needs, is inductive, and supports the inclusion of a priori and emergent codes. In this case, our goal was to capture information on psychotropic oversight approaches and their respective implementation and timeline. Framework analysis includes 5 systematic and visible steps so that collaborators understand the results and their interpretation. Our team engaged in each of the 5 steps in analyzing data, specifically (1) becoming familiar with data, (2) developing and testing an initial codebook based on interdisciplinary team meetings for 6 transcripts, (3) coding the remainder of the data with the final codebook, (4) charting the data by state to gain a deeper understanding of the information captured for each approach, and (5) interpreting the data.¹¹ New information that could not be appropriately coded within the existing codebook framework was dealt with using a consensus-based decision-making approach, so that new codes were added to the existing codebook that further enhanced it. Data from respondents in a particular state were charted to document the totality of mechanisms in place for a specific state.

For data derived from document and legislative review, we employed content analysis, composed of both a priori and emergent codes.¹³ A priori codes were derived from federal guidance on psychotropic medication oversight and extant literature. Emergent codes were identified based on an initial review of documents, including identification of characteristics that varied in each of the 11 domains between states (see “Strategies used in the 4 states for AP oversight” section). A priori and emergent codes were developed into a codebook that coders applied to textual data. The findings were then entered into state-specific matrices to systematically document both the mechanisms and characteristics of each mechanism.

Data charted for all respondents from a specific state were then transformed into state-specific narrative summary sheets with considerable detail on the evolution of policies in multiple relevant domains over more than a decade (typically in the range of 8-10 pages); these summary sheets were validated by state key informants. Each detailed state-specific summary along with the implementation timeline was sent to key informants within the respective state to confirm and update the findings. The key informants were encouraged to forward the summary sheets to individuals who were well positioned to provide additional information in the requested areas. The research team revised the summary sheets and timelines accordingly, based on feedback from the informant's data validation efforts.

Aim 2 Stakeholder Perspectives and Patient-Reported Outcomes

Within aim 2, we conducted qualitative data collection with key stakeholder groups that were integrally involved in states' psychotropic medication oversight mechanisms: prescribers/clinicians, foster care alumni, foster caregivers, and child welfare caseworkers. We aimed to assess how stakeholders understood the state-specific monitoring mechanisms (knowledge); the extent to which they had been affected by these mechanisms (experiences); and how they navigated (or recognized) the tensions inherent to these mechanisms (trade-offs). As we collected pilot data, we found that few stakeholders other than state policy leaders were fully aware of the components of state monitoring systems in general. What resonated with them as a priority were those particular mechanisms that were relevant to their experience. As confirmed by the SAB and stakeholder partners with lived experience, the most salient structure to the stakeholder groups overall was informed consent to treatment; thus, for them, this was the primary monitoring mechanism of interest. The 4 study states implemented informed consent similarly (thereby streamlining our analysis across the states and across stakeholder groups). Informed consent was also the mechanism that all participating stakeholders were the most eager to discuss because of its relevance to their system experience. Because shared decision-making is also inextricably linked with informed consent, we present our analysis of shared decision-making in this report.

Aim 2 Participants

Three of the stakeholder groups—prescribers, caregivers, and caseworkers—were recruited from the 4 states in the PCORI study^* (Ohio, Texas, Washington, and Wisconsin), with the assistance of advisory board members or state officials. The foster care alumni, young adults who had experience with AP medication treatment while in foster care, were recruited through nationally based advocacy groups and therefore reflected a broader geographic region and not just the 4 study states. All participants completed informed consent procedures that were approved by the university's IRB. With the exception of the caseworkers, all stakeholder respondents received an Amazon gift card following the conclusion of their participation in the project. According to states' regulations, caseworkers are prohibited from receiving compensation for participation in research projects. Depending on the stakeholder group, participants were either individually interviewed with a semistructured interview guide or were involved in a web-based focus group using a deliberative discussion format.

For aim 2, with 1 exception (described below), our recruitment approach was the same as that originally proposed in our grant application. Through a purposive sampling design, our approach was to identify potential participants via our state partners, some serving on our SAB and others referred by the SAB. Through these referrals, we conducted outreach to key senior representatives of multiple state agencies who provided us with contact information for potential participants.

To screen interested stakeholders and determine their eligibility, we employed 2 strategies: 1 for foster care alumni and 1 for all other stakeholders. For all stakeholders except foster care alumni, our project coordinator conducted individual phone interviews with each interested individual. Screening interviews consisted of approximately 18 to 20 questions pertaining to demographic information, employment history, and background of working with youth in foster care who have been prescribed AP medication. Each individual survey was documented in Qualtrics (https://www.qualtrics.com), and then transferred into SPSS (https://www.ibm.com/analytics/spss-statistics-software) for descriptive analysis.

Recruitment of the foster care alumni required a prescreening process. This was determined to be important by Youth MOVE National partners due to the sensitive topics under discussion and because many of the potential participants have experienced trauma. In our recruitment approach with young adults, we worked closely with our Youth MOVE National partners to develop a prescreening protocol that was sensitive to these traumatic experiences and would avoid additional trauma. A senior member of Youth MOVE National was the initial point of contact for every individual interested in participating in the project; Youth MOVE National's phone number and email address were the primary contact information on the recruitment flyers. Youth MOVE National acquainted each individual with the project and proceeded with the screening interview, which was lengthier than the version for the other stakeholder groups, so that Youth MOVE National and the prospective respondent could assess whether each individual was prescribed AP medication (and not just psychotropic medication) and that they met other critical eligibility criteria (Table 1). Youth MOVE National also assessed whether the individual was comfortable with emotional disclosures related to their mental health treatment in small-group settings. Youth MOVE National then worked closely with the CHSR team to discuss the screening of each individual and whether CHSR should reach out to potential recruits to discuss informed consent for the project and to schedule participation in the online focus group. The information for both the prescreen and the screen were captured in a Qualtrics database. Very few caseworkers, prescribers, and caregivers were determined to be ineligible for the project. For alumni, we declined further participation for individuals who were determined not to be in foster care during the time frame and who did not receive AP medication treatment based on their self-report after reviewing with the prescreener a comprehensive list of AP medications.

Table 1

Inclusion/Exclusion Criteria for Aim 2.

For prescribers, the SAB advised that we interview clinicians who had robust experience with the monitoring systems in the last 6 months. These clinicians were characterized by the SAB as “high-volume prescribers” relative to their state peers. A list of clinicians was developed from each state's Medicaid administrative data. We also identified high-impact areas (ie, geographic regions in each state in which AP medication prescribing to children in foster care with mental health needs was prevalent) and sent invitation letters individually addressed to potential respondents. The recruitment letter was customized for each state and was co-signed by the principal investigator (PI) (Dr Stephen Crystal) and co-PI for aim 2 (Dr Cassandra Simmel) as well as key state partners (representatives from Medicaid and child welfare agencies). We followed up with individual phone calls, emails, and faxes to each potential participant to inquire about their response to our invitation letter and to acquaint them with the study. We then initiated a screening interview with each potential participant to ensure that they met the enrollment eligibility criteria. As the lists of high-volume prescribers were exhausted, we systematically moved on to individuals with less robust experience, representing a second “wave” of recruitment. Overall and across states, prescribers minimally served between 5 and 10 children in foster care during a 6-month time frame.

For caseworkers, we employed a similar strategy; state partners helped us identify potential participants, and we sent invitation letters to these participants, followed by phone calls to inquire about their interest and availability. If interested, we proceeded with a screening interview to determine eligibility. Foster care “cases” are distributed differently in different states. In Texas, for example, a special team of highly experienced caseworkers covers all the children with mental health needs placed in foster care. Another state has regional networks of caseworkers who are part of a private subcontract with local case management organizations. We worked with partners at the child welfare agencies to contact the caseworker respondents.

For foster care alumni, we worked closely with our partners at Youth MOVE National and the FCAA to conduct outreach and enhance recruitment efforts. In early 2017, two senior Youth MOVE National representatives were added to the Rutgers IRB protocol to help us in this effort. This strategy has involved posting flyers on Youth MOVE National's social media sites (eg, X [formerly Twitter] and Facebook) as well as holding joint appearances between CHSR team members and Youth MOVE representatives at national conferences to advertise the project. Youth MOVE National and NFPA also sent targeted emails and flyers to their many local chapter offices. FCAA includes many individuals who are currently foster parents and/or Court-Appointed Special Advocate (CASA) members; due to their unique experiences, these individuals provided the study with a broader pool of potential respondents. Our approach thus deployed many independent resources for reaching potential participants. Initial caregiver recruitment through our SAB members was not successful, leading to collaboration with the NFPA and its state affiliates. Local champions were identified and relationships established to support state-specific caregiver recruitment.

As with aim 1, due to extensive respondent engagement efforts in partnership with stakeholders, very few individuals declined further involvement once enrolled in the project. Eight alumni declined due to personal difficulties and inability to attend the online focus groups. We tried several times to engage them and enroll them in a different focus group and even offered to schedule the groups at multiple times (eg, evenings and weekends) to accommodate various conflicts, but ultimately, this did not work. We learned during this project that the alumni stakeholder group is particularly challenging to schedule given their often-complicated life circumstances; however, by providing a great deal of flexibility in scheduling, following up energetically, and collaborating with Youth MOVE and other advocacy organizations, we succeeded in eliciting the perspectives of this important stakeholder group.

Aim 2 Analytic Methods: Qualitative Stakeholder Data Analysis

Data analysis included descriptive summaries of stakeholders' demographic and background information obtained from surveys. Interview and focus group transcripts were imported into Dedoose (https://www.dedoose.com/) and analyzed by a trained, multidisciplinary research team. The analysis of interview and focus group transcripts included many steps to ensure the rigor of the qualitative methods.

A codebook was created to organize the qualitative data and support analysis. To develop the codebook, members of the team reviewed a sample of transcripts from each stakeholder group. Over a series of meetings, team members discussed ideas about categories of information to capture through the coding process. These ideas consisted of an a priori set of codes and emergent codes. A draft codebook was created based on a review of these initial transcripts. Two members of the research team then independently tested the codebook with a second sample of transcripts, and the entire team met to discuss discrepancies. This process continued with each stakeholder group until the team felt the codebooks were comprehensive. Interrater reliability was also measured for a subset of the stakeholder interview transcripts, ranging from 0.74 to 0.89 across all stakeholder groups and analysts. The team reached a consensus on codebook definitions and applications and then proceeded with coding the remaining interviews. Team members met regularly to discuss their coding process and resolve any questions. While codebook revisions took place for each stakeholder group, the a priori codes remained throughout. Thematic analysis was completed by observing code application within and across groups. The research team met to discuss themes and sensitizing concepts that emerged across the groups as they pertained to the oversight mechanism of informed consent and the concept of shared decision-making.

Aim 3 State-Specific Medicaid Administrative Data Analyses

We did not recruit respondents for aim 3, but rather used deidentified Medicaid administrative data to examine stakeholder-prioritized outcomes; thus, this aim does not include any respondent recruitment. This approach allowed us to examine data on the entire population of children and youth receiving Medicaid based on foster care status, as well as relevant comparison groups, avoiding bias from selective participation of respondents. Although the inclusion criteria and comparison group definitions varied by measure and analysis, in general, we identified children in foster care with Medicaid basis of eligibility codes and examined outcomes for children and youth under 18 years of age who were enrolled in Medicaid for at least 11 months during 1-year analytic periods, excluding those who showed any dual Medicare eligibility during the period.

Three states submitted their data directly to CHSR for the analysis. For Washington State, we used Medicaid Analytic eXtract (MAX) data licensed directly from CMS. The protocol was modified to use a different source for the state's claims data (from CMS via MAX rather than directly from the state) for multiple reasons: Delays in the state's process for approving data release would have delayed the overall project timetable, and the use of MAX had scientific advantages because it allowed for comparison with multiple states that did not enact mandatory review policies.

Aim 3 was designed to understand the impact of state oversight programs on quality measures prioritized by stakeholders, patients, and families. We constructed measures of AP use based on filled prescription data from the Medicaid claims. A list of qualifying AP medications is included in the appendix. The selection of measures of safe and judicious management of AP prescribing reflected input from the measures prioritization analysis described in the Results section, from the SAB, and from other measurement development activities that incorporated the input of diverse stakeholders to determine measures important to patients and stakeholders; this included the work of the Medicaid Medical Directors Learning Network (MMDLN)/Rutgers Center for Education and Research on Mental Health Therapeutics (CERT) Antipsychotics in Children Project and the National Collaborative for Innovation in Quality Measurement (NCINQ), whose work led to nationally endorsed HEDIS quality measures of safe and judicious pediatric prescribing of AP medications. Candidate measures were identified from the MMDLN/CERT project, the MEDNET project, HEDIS NCQA, and the NCQA NCINQ project, with an emphasis on measures that had been tested and had achieved broad acceptance in use for quality improvement. To the extent possible, we aligned measurement construction with nationally endorsed HEDIS quality measures. Key measures included metabolic monitoring of AP-treated children, avoidance of AP polypharmacy (APC), and the use of first-line psychosocial intervention among AP-treated children. The selection of state interventions for the study and the metrics most relevant to the aims of these interventions were based on the work of aim 1 and on input from the SAB and considered the relevance of the intervention to potential adopters in other states; the policy significance of the model being tested; and the evaluability of the intervention through observational analysis of claims data (eg, taking into account whether interventions took place at a clearly defined time point, involved clearly defined policy/administrative changes of significant scope, and targeted specific practices identified as important by stakeholders and patients and for which appropriate comparison populations were available). Metric specifications and inclusion criteria are summarized in the appendix. We used standard statistical methods for analysis of the claims data, including logistic regression and difference-in-differences (DID) modeling, adapted as appropriate to each specific analysis. Although the final narrowing down of measures was informed by the SAB and the prioritization survey, all of the measures used were prespecified and represent widely accepted metrics used in quality assessment. Based on aim 1 work on the characterization and analysis of state oversight initiatives, and their timing and implementation, the following initiatives were selected for pre-post analysis of claims-based outcomes under aim 3 (Table 2).

Table 2

Summary Table of State Interventions Hypothesized to Improve Safe and Judicious Prescribing of AP Medications.

Illustration of previously described methods: application to specialized MMCO intervention in Texas: a case study

The methodological approach described previously can be illustrated by our analysis of the effect of implementing a multimodal specialized MMCO initiative in Texas, which was identified through aim 1 work, the SAB, and other sources as being influential for other states, with many other states having either implemented similar models recently or seriously considered future adoption of this model. The methods and results on this topic have been published in Mackie et al.¹⁴ This paper evaluated a multimodal intervention designed to accommodate the special health care needs of Medicaid-insured youth in foster care. The intervention was developed by the State of Texas and implemented by Superior HealthPlan in 2008. In 2007, AP prescribing rates among youth in foster care in Texas were among the highest rates nationally (21.7%).⁶ Most of this prescribing was done by psychiatrists.

Analytic methods as applied to Texas MMCO intervention

In response to concerns regarding AP use among youth in foster care within this context, Superior HealthPlan implemented a multimodal intervention available only to Medicaid-insured children in foster care that included 4 elements: (1) a mental health screening (a customized version of the Child and Adolescent Needs and Strengths assessment¹⁵) administered within 72 hours of removal from the caregiver of origin to another home or residential care setting; (2) a health passport (ie, a structured report from claims-based data designed to make these data usable by clinicians to understand treatment history in a summarized fashion); (3) an elective psychiatric consultation line available to child welfare staff, caregivers, and judges; and (4) a retrospective drug use review. Details of the implementation of these components were characterized with document review and key policy informant interviews under aim 1.

First, the mental health screening tool was an assessment instrument that included surveillance questions to identify behavioral health needs, including questions related to whether youth have been diagnosed with a mental health condition, symptoms of depression and anxiety, and prior service use. The screening tool was implemented for the purpose of identifying needs for mental health services. The results were entered into a health passport, followed by a comprehensive behavioral and mental health screening administered within 30 days by certified clinicians.

Second, the Texas Family Code (section 266.006) required the state child welfare agency to establish and maintain a health passport for each child in foster care. The Texas “health passport” is a 2-year retrospective health history given to foster care guardians for children with claims histories in Medicaid. The MMCO implemented a health passport that pulled in 2 years of Medicaid-claims data for youth enrolled in Medicaid or the Children's Health Insurance Program before foster care entry (about 52% of children entering care). Access to the health passport was available to medical consenters, CASA staff, authorized users for out-of-home treatment settings, health care providers, and child welfare staff.

Third, the MMCO implemented certified psychiatric consultation services available for requests from the court and other stakeholders (eg, caregivers, caseworkers). Although data regarding implementation are not available between 2008 and 2010, quarterly records of implementation from 2016 to 2018 suggest higher rates of use from other stakeholder requests (range, 9-53 requests/quarter) than from court requests (range, 3-14 requests/quarter).

Finally, the retrospective drug use review assessed whether psychotropic medication administered to youth in foster care was consistent with the Psychotropic Medication Utilization Review Parameters for Foster Children.¹⁶ Quarterly reviews were conducted by master's-level professionals at a behavioral health unit, working under supervision of a clinical pharmacist and child and adolescent psychiatrist. If prescribing practices were not consistent with the parameters, a clinician reached out to the prescribing provider to discuss the case and provide necessary follow-up.

Corroborated by other national studies of state AP oversight initiatives,^17-19 in aim 1, we conducted both legislative review and key informant interviews and found no evidence of AP oversight initiatives implemented during the study period for Medicaid-insured youth who were not in foster care, including children who were adopted.¹⁸

To examine the effects of implementation, we conducted a retrospective observational study of intervention outcomes using Medicaid administrative data. We examined the impact of intervention with a DID design; outcomes of interest were compared before the intervention (2006-2007) and after implementation (2009-2010) for Medicaid-insured youth in foster care. The comparison group comprised Medicaid-insured youth eligible through adoption assistance (ie, adopted youth). Our use of a comparison group of adopted youth allows for our estimates of effectiveness to account for secular trends (eg, greater awareness of potential cardiometabolic adverse effects associated with pediatric AP prescribing) that would presumably influence AP prescribing and/or the annual prevalence of monitoring cardiometabolic adverse effects (lipid and glucose testing) among Medicaid-insured youth in adoptive care and foster care equally and independent of the MMCO innovation.²⁰ Outcomes representing AP use and quality included (1) annual prevalence of any dispensed AP, (2) annual prevalence of youth dispensed APs for at least 90 consecutive days, and (3) annual prevalence of youth dispensed APs with glucose and lipid testing. This study was approved by the Rutgers IRB.

To examine the time period of program implementation (which preceded the dates of the data provided to us by the state), we used MAX data from CMS.²¹ We constructed an analytical data set representing annual personal summaries of service use (eg, glucose and lipid testing) and diagnosis histories for each youth for the years 2006 and 2007 (data points observed before implementation) and for 2009 and 2010 (postimplementation observations). Multiple years of data were pooled as repeated cross-sections. We identified whether any or continuous AP dispensing and glucose and lipid testing took place for each person-year. Within this person-year framework, an individual could contribute to more than 1 person-year across the 4 years (ie, 2006, 2007, 2009, and 2010).

Our statistical approach first examined whether trajectories of AP dispensing and glucose and lipid testing for youth in foster and adoptive care were comparable before the intervention exposure. Potential differences in prescribing patterns may exist between these 2 populations given the inherent differences in the stability of their family situations.²² For example, the sample of youth in foster care might disproportionately include young people who “age out” of the child welfare system. Youth who “age out” of foster care into independent living differ in several respects from those who are adopted in that they are more likely to be unemployed,²³ have unplanned pregnancies,²³ or become arrested or incarcerated.²⁴ Additionally, secular trends in AP prescribing and glucose and lipid testing may also be affected by several relevant high-profile articles released during the proposed study period, most notably the American Academy of Child and Adolescent Psychiatry (AACAP) statement on Pharmaceutical Benefits Management,²⁵ reviews on AP treatment for disruptive behavior disorders for children and adolescents,²⁶ and monitoring and management of AP-related metabolic and endocrine adverse events.²⁷ However, although the baseline rates for these populations may differ, they would likely be affected by general secular trends in the state in similar ways. To investigate whether trends were comparable for youth in foster care and adoptive care for each of 3 outcomes, we tested for parallel trends from 2006-2007 and found no significant differences in trends between the exposure and comparison groups for each of the study outcomes. When preexisting trends were comparable, DID models demonstrated the conditions necessary to control for secular trends (such as evolving evidence on metabolic adverse effects of AP medications) that would be expected to influence both the intervention and comparison populations.²⁰ We then estimated DID models comparing the change in the group exposed to the intervention (in this case, youth in foster care) with a group not exposed to the intervention (in this case, adopted children). Further details of the statistical model are included in the “Results” section to simplify the interpretation of the reported results. We applied this general model across each of the states and interventions studied, although the general method and comparator populations were adapted to state variations in monitoring systems. For example, for the Washington analysis of the Medicaid prior authorization program, we used a synthetic cohort approach. This approach was selected as being best suited to this analysis because it allowed us to use a multistate comparator of states that did not implement similar prior authorization interventions. Because the Washington intervention was statewide, the appropriate within-state comparators were not subject to the policy innovation and were not available. The opportunity to use this approach became available as a result of acquiring the 45-state MAX data through 2012.

Results

By virtue of their large scale and the complicated systems within which they are embedded, state initiatives such as those we assessed are complex, often multimodal, and generally adapted and modified during implementation. Studying interventions at this scale and complexity constitutes an unusually complex research challenge to which the study's mixed-method approach is well suited. This makes the problem less straightforward than in studies that examine interventions that are more clearly structured and less evolutionary, involving fewer intervention components, and/or implemented at a more limited number of sites and settings. Thus, our study may be somewhat atypical in the PCORI portfolio in its breadth and necessarily sequential approach, with the final selection of initiatives and measures informed by initial work under the first 2 aims. However, this was a necessary consequence of the scale, complexity, and multimodal nature of the state oversight systems. Complexity is inherent in the research topic, and studying these large-scale state-level interventions offers the potential to inform the improvement of systems that powerfully affect hundreds of thousands of children across thousands of settings.

Aims 1 and 2

Aim 1. Key Informants for State Policy Analysis

Thirty-eight key informants were selected based on their knowledge of the policy and programmatic mechanisms for psychotropic medication oversight in their state. Table 3 shows the distribution of the 38 respondents by state, education, and departmental affiliation, including 10 supplemental respondents added to provide more information on county-level policy and program variations in Ohio. Key informants were recruited through a snowball sampling approach, with input from SAB members who identified individuals who informed the development and/or implementation of oversight systems. At a minimum, this included at least 1 key informant from state Medicaid, child welfare, and mental health agencies. Key informants from other state or county agencies were also sampled depending on the configuration of the oversight.

Table 3

Aim 1 Key Informant Sample of State Policy Makers, Midlevel Managers, and Partners (N = 38).

Aim 2. Participant Results: Broad Qualitative Stakeholder Data Collection

Four stakeholder groups were identified as critical actors and sufficiently knowledgeable respondents in ensuring quality mental health treatment for children in foster care and the prescribing of AP medications (Table 4).

Table 4

Aim 2 Respondent Demographics by Stakeholder Type.

Aim 1. State Oversight Strategies

Strategies used in the 4 states for AP oversight

With SAB input, aim 1 extended prior frameworks on elements of psychotropic medication oversight programs. Consistent with prior studies,¹⁸ our taxonomy (see Figure 1) illustrates that monitoring mechanisms may occur before medication dispensing (ie, prospective) or after dispensing (ie, retrospective). States also invest in other supports, including psychiatric consultation lines, quality improvement initiatives, trauma-informed systems, care coordination, and multimodal initiatives. Our SAB recommended the inclusion of quality improvement collaboratives, trauma-informed initiatives, and care coordination to our initial taxonomy of monitoring approaches. Consistent with aim 2 findings, SAB members emphasized that these mechanisms are part of a robust monitoring approach that addresses the multiple barriers to optimal psychopharmacological prescribing.

Figure 1

Taxonomy of Monitoring Mechanisms for Psychotropic Medication Oversight Systems.

Aim 1 was an in-depth investigation of the evolution of psychotropic oversight programs for children and adolescents in foster care across the states. From 2002 through January 2017, state public sector systems across the 4 states implemented 68 new oversight mechanisms. On average, 38% of these (n = 26) were revised during this 15-year period. States most frequently initiated new system-wide service arrays for trauma-informed psychosocial therapies (21%[n = 14]) and new protocols for routine behavioral and mental health evaluations for children in foster care (15% [n = 10]). The greatest number of revisions were implemented for administrative case reviews (23% [n = 6]), behavioral and mental health evaluations routinely administered to children in foster care (23% [n = 6]), and trauma-informed behavioral and mental health services that seek to integrate knowledge about traumatic experiences and to actively resist episodes of retraumatization (5% [n = 5]). Respondents were less likely to report revisions of other AP medication prescription monitoring mechanisms once initiated.

The aim 1 findings also identified key implementation trends. First, early experiments that created specialized Medicaid managed care plans to specialize in services for children in foster care are increasingly relevant. In each of the 4 states, managed care arrangements were either under development or in place for all or a subset of children in foster care. As children in foster care were historically enrolled in fee-for-service systems to facilitate flexibility in provider and service access, investigation of the multimodal tools available to managed care arrangements arose as a critical policy lever for further examination. States also sought creative solutions to mitigate potential unintended consequences of AP monitoring programs. First, key informants reported incremental approaches to the expansion of psychotropic monitoring programs. They frequently cited the importance of having invested in supports before implementation of a prior authorization program or other means that might restrict access to needed treatment; for example, key informants in Washington reported implementing a psychiatric access line nearly a year and a half before implementing a mandatory peer review, establishing collegial consultative relationships and education efforts before mandatory reviews. The Washington State prior authorization provided a 60-day authorization window allowing for temporary medication administration to occur during the period in which the prescription recommendation is being reviewed by a clinical peer. In other states, the prior authorization program instead included an “emergency window” in which prescribing could occur with a special authorization. Given our interest in whether prior authorization was as effective with the 60-day authorization window, our aim 3 work also examines the effectiveness of this incremental approach taken in Washington State.

During our initial work in aim 1, we found extensive heterogeneity by county in Ohio. We purposefully selected 3 counties (Franklin, Lorain, and Lucas) as well as the county consortia selected for Ohio MM Collaborative²⁸ implementation; we conducted 10 additional key informant interviews and reviewed county-specific documents and legislation to investigate variation. Our analyses found extensive county-level innovation supporting county-specific investigation into implementation within Lucas, Lorain, and Franklin counties and the multicounty Ohio MM consortia. A review of the service use in each region suggested that service resources and systems differed considerably across regions. We identified 2 types of configuration. In 1 configuration, a high volume of youths were served by general pediatricians in large-volume practices and tertiary children's hospitals and by a few high-volume prescribers from community behavioral health agencies. In other counties, the pattern was that psychotropic medication was prescribed mostly by psychiatrists in community mental health and residential treatment centers (and a few general pediatric providers) working with youths with very complex conditions. As a result, each regional collaboration organized itself differently to bring together the clinical, educational, child welfare, and other social service entities specific to the region. Ohio counties varied in the services in which they invested during this period of time. While Franklin and Lorain especially invested in the identification and treatment of trauma to expand trauma-informed care approaches, respectively, Lucas centered its efforts on developing a more robust approach to psychotropic monitoring and quality improvement.

Ohio MM offered the opportunity to examine a multimodal learning collaborative implemented across a series of counties and targeted specific settings where prescribing concerns were greatest. While study power constraints ultimately limited our ability to perform claims-based analyses for individual counties (because base rates of AP use were lower than anticipated, cell sizes for metrics calculated only on AP-treated children were too small for quantitative statistical analysis), the added county-level key informant interviews and document review provided vital context in interpreting the overall outcomes associated with Ohio MM in pooled data across the counties where it was piloted and in the rest of the state. This information also provided important insight into the extent of county-level variation in key oversight processes and in the informed consent and shared-decision-making process, confirming information learned from aim 2 stakeholders about this wide degree of variation. These findings suggest that in states with county-level child welfare administration, the effectiveness of state-level interventions to improve adherence to AP management by implementing best practices depends significantly on the engagement of county-level decision makers and county-level prescriber and child welfare communities in these initiatives. In Ohio, this wide county-level variation significantly complicated the challenge faced by MM as a state-initiated improvement strategy. In states where services are administered at the county level and supervised at the state level, the challenge of quality improvement is linked to the capacity at the county and state levels to communicate and coordinate their oversight initiatives. In this regard, the documentation of county-level initiatives related to psychotropic drug–prescribing oversight was of great interest to the project's state partners and contributed to the development of a collaborative relationship with these state-level stakeholder partnerships that was essential to the study, exemplifying the reciprocal sharing of information that is vital to such studies.

Implemented on September 30, 2014, Ohio's Franklin County Children's Services (FCCS) designated consenting authority for nonstandard treatment, including the use of psychotropic medications, to a registered nurse (RN) housed within the medical services unit. The FCCS RN reviews requests based on criteria in guidelines provided in the Psychotropic Medication Toolkit for Public Children Services Agencies.²⁹ The RN may consult with the nurse supervisor, the prescriber, the caseworker, or the contracted child psychiatrist. Supported by the Administration for Children and Families, FCCS also implemented the Gateway Consultation, Assessment, Linkage, and Liaison Project to provide trauma-informed screenings and assessments for children involved in child welfare systems within FCCS. For this project implemented in February 2015, FCCS caseworkers routinely administer a screening for trauma based on the child's age. The results of the screening are subsequently scored by clinicians from Nationwide Children's Hospital (NCH) in Columbus, Ohio. If concern is indicated by the score on the screening tools, the child receives a trauma assessment conducted by an NCH clinician. Based on the results of the screening and assessment for trauma, the assigned FCCS caseworker will refer a child and their parent(s), with indicated concerns, to a community provider for ongoing trauma-informed services.

During our study period, Ohio's investments in psychosocial therapeutic services occurred in Lucas County through an innovation grant received in January 2014 that supported county-wide training in cognitive processing therapy for caseworkers as well as intergenerational treatment and “trauma competency” trainings for caregivers. Lorain County Children's Services (LCCS) implemented a psychotropic administration request (PAR) form in January 2016; the form was required by policy to be completed before consent for any psychotropic medications prescribed to children in foster care. In June 2017, the PAR form was incorporated into a quality improvement process in which the continuous quality improvement (CQI) coordinator uses data from these forms to provide monthly estimates to nurses co-located in child welfare agencies to confirm accuracy and inform educational efforts targeted to caseworkers. In September 2017, LCCS established a psychiatric consultation line available to the CQI coordinator and executive director on an “as-needed basis” for consultations during the PAR.

Finally, in September 2012, the Ohio Department of Medicaid launched a quality improvement initiative called Ohio Minds Matter to promote the safe and appropriate prescribing of AP medications to youth.²⁸ This program had several objectives that were disseminated across the state, including a 25% reduction in the use of the following: AP medications among children aged under 6 years, 2 or more AP medications for longer than 2 months (APC), and 4 or more psychotropic medications in youth under 18 years of age. This investment targeted providers of services to Medicaid-eligible children, including those in foster care. Thirteen counties in Ohio were targeted to implement this initiative. A clinical advisory panel consisting of experts in pediatrics, psychiatry, and pharmacology identified the objectives listed previously and developed evidence-based guidelines for AP medication prescribing. The Ohio MM initiative included online modules, fact sheets, and shared-decision-making tools; routine performance benchmarks; and academic detailing for prescribing clinicians. The counties selected for MM implementation were organized into 3 multicounty consortia, in which provider education efforts were concentrated and provider coalitions were formed. However, the MM tool kits and educational materials were also made available statewide.

Aim 2. Stakeholder Responses and Prioritization of Measures

Informed consent and shared decision-making: alumni and other stakeholder perspectives

While aim 1 characterized how policy makers believed informed consent was authorized, aim 2 provided clarity about how stakeholders view the informed consent process during a medication-oriented mental health clinical encounter. Insights were particularly informative on these issues. Informed consent and shared decision-making (assent) in consent for medication use were seen by all stakeholder groups as issues of fundamental importance whose complexity can defy simplistic rules about the roles of participants, given the great variation in circumstances. In the assent process, youth and young adults of widely ranging age and maturity must grapple with proposed treatment involving powerful medications that may improve self-regulation of emotions and behavior but that are accompanied by significant risks and adverse effects. Informed consent for children in foster care is further complicated because rules and procedures concerning which individuals or agencies have the legal responsibility to provide medical consent vary a great deal between and within states. As well, stakeholders hold different roles in the process, and these are not always clearly delineated. For example, we were told that in some counties in Texas, a judge is responsible for “medical consent,” while in other regions, the state courts rely on the caseworker to determine whether the medication is warranted. Who ultimately provides medication consent emerged as a vital issue for each of the stakeholder groups, with a notable amount of agreement across stakeholder groups and across states. Among respondents, there was a considerable degree of consensus that consent/assent is a complex issue that requires individualization and for which simple, formulaic rules governing who can consent are problematic.

The results of our aim 2 analyses indicate that all stakeholder groups valued a team-based approach to the process of consenting for medication use and believed this was necessary for authentic informed consent to unfold. Moreover, there was agreement about the benefits of having many stakeholders involved, especially when they collectively provided multiple perspectives on the specific case. Study participants believed the informed consent and shared-decision-making (assent) processes were means to provide checks and balances across the mental health care system, the medical system, and the child welfare system. However, while a great deal of respect for the team-based approach and each participant's role was reflected in the interview data, interviewees expressed frustration about the difficulty of these decisions given the need to work across complex systems with multiple stakeholders. A common source of frustration involved concern about the quality of decisions made by individuals who were not present at the clinical encounter where the clinician recommendation for psychotropics arose and who had varying degrees of understanding of the individual situation. Requirements for multiple levels of review were seen by some respondents as a protection but by others as a potential source of undue delay and possible harm to children in need of immediate assistance. As noted previously, although each stakeholder group manifests different roles in the informed consent/shared-decision-making process, there were many similarities in what they valued in the process of eliciting informed consent as well as a general recognition that their roles were genuinely distinct from one another. There was a broad consensus that optimally, the informed consent process should reflect well-informed understanding by decision makers of youths' histories, current situations, assessment of psychological symptoms, and present complaints and problems. The difficulty of achieving this was generally acknowledged. Respondents noted as challenges, for example, the chaotic lives of many children in foster care and the fact that clinicians are often asked to make medication decisions during a visit with a child whom they have not previously met during a crisis-induced clinical encounter. Respondents noted that decision-making often takes place in circumstances where full information on clinical history and an established therapeutic relationship are lacking. While acknowledging the importance of good communication and shared decision-making, caregivers and caseworkers noted inherent knowledge and power dynamics among clinicians and youth, which can inhibit effective communication. Clinicians emphasized the need for communication and informed consent to happen in person during appointments, and they were often frustrated by mandated authorities (who were often in the contiguous child welfare system and not the medical system) not being present or not responding in a timely manner.

Moreover, while there was an acknowledgment of the lengthy time and delays involved in obtaining informed consent, there were differing perspectives on whether such delays were a positive or negative aspect of the process. Problems that were often encountered by alumni included insufficient time and information provided by the prescriber to allow for genuine informed consent/assent, decisions made by clinicians with whom no real relationship had been established, and the role in the consent process of adults not closely involved with the child's current life and who were not knowledgeable about the child's circumstances and concerns. For example, concerns were expressed that in situations where judicial approval was required, judges were not sufficiently knowledgeable about the child's individual circumstances and the clinical issues involved in the decision. Clinicians and youth generally placed more value on youth agency and comprehension during the informed consent process than did other stakeholder groups. Caseworkers and caregivers emphasized their roles as protectors of youth and as stakeholders who knew the youth best. Despite these differences in emphasis on who could best represent the interests of the child, members of each group generally acknowledged the concerns of the others. For example, while clinicians and caseworkers highlighted the wide differences in maturity and decision-making ability among children in care, alumni also recognized this as an important issue. Conversely, while alumni especially emphasized the importance of youths' involvement in decision-making, this was also generally recognized as an important principle by clinicians and caseworkers.

Identifying policy outcome measures most important to stakeholders: best-worst scaling choice experiment

A key challenge in patient-centered comparative effectiveness research is the identification and prioritization of outcome measures—in this case, outcomes influenced by policies and state interventions—that are important to patients and other stakeholders. In our study, we informed our selection of outcome measures through systematic elicitation of stakeholder priorities, complementing input from the SAB, and other measurement development activities that incorporated the input of diverse stakeholders to identify outcome measures that are derived from administrative data and are important to patients and stakeholders. To do so, we adapted methods used in marketing research to elicit priorities of stakeholder group claims-based outcomes (Table 5). Key stakeholder groups including policy makers (n = 31), foster care alumni (n = 28), prescribing clinicians (n = 32), caseworkers (n = 23), and caregivers (n = 18) completed a best-worst scaling (BWS) choice experiment. Stakeholders received a survey that included a scenario on AP monitoring programs and were asked to rank relevant claims-based metrics as most and least important to an evaluation of the program. These candidate items were selected from measures emerging from prior quality improvement collaborative efforts and prior research, with a focus on measures that had achieved broad acceptability, such as those captured in the NCQA HEDIS AP measurement suite. These HEDIS measures have been vetted by CMS and used as performance measures across MMCOs nationally.

Table 5

Relative Outcome Prioritization Across Stakeholders.

Descriptive statistics were used to summarize respondent characteristics by stakeholder group. We employed a hierarchical Bayes multinomial logit model to calculate utility scores and 95% CIs for each of the 11 candidate measures and present findings as ratio-scale probabilities (0-100). The BWS analysis showed that safety indicators for AP use ranked highest, including AP use among young children, AP co-pharmacy, and cross-class polypharmacy. Foster care alumni and caregivers prioritized psychosocial treatment as a first-line treatment. Monitoring for adverse effects, such as metabolic monitoring, was prioritized at an intermediate level. Unintended consequences of implementing an AP monitoring program ranked lowest, including potential substitution effects, psychiatric hospital stays, and ED use. Overall, the BWS choice experiment resulted in a rank-ordering of standardized scores for outcome metrics. Particularly salient in patient and stakeholder responses was the high priority generally assigned to process measures of safe use related to the circumstances and management of prescribing, such as measures of polypharmacy and AP prescribing among very young children, and the lower priority assigned to measures aimed at capturing distal effects of treatments, such as psychiatric hospitalization and ED use. This input was consistent with input from the SAB and concerns by investigators that efforts to detect such effects would be less rigorous and more subject to confounding factors such as changes in bed supply and payment issues. Based on this information in combination with scientific considerations and evolving literature on AP medication risks, the results from aim 1, and the results of other stakeholder-advised measure development processes such as those of the NCINQ,³⁰ we focused on AP use rates by child's age, receipt of first-line psychosocial treatment, metabolic monitoring, and APC as the focus for analysis of policy and program outcomes. Overall, we concluded that program monitoring, evaluation studies, and policy development for psychotropic oversight programs should consider prioritizing indicators that align with the preferences of stakeholder groups, and that BWS represents a promising, albeit underused, approach to prioritization that deserves consideration in patient-centered comparative outcomes research. Measures intended to capture unintended consequences of implementing an AP monitoring program, such as increases in psychiatric hospitalization, were ranked lowest by our stakeholders. The SAB members raised concerns about the validity of these measures, noting that in their states, rates of hospitalization and ED use over time were influenced by multiple supply, demand, and policy factors that were difficult to measure and that would confound efforts to attribute changes to the effects of oversight policies. For example, a child in crisis may only be offered inpatient treatment if there is a bed available. Regional differences in system access and resources are well documented. This input reinforced scientific concerns about the suitability of the available measures for detecting the effects of oversight policies. As a result, these measures were not included in the aim 3 analyses. We did, as proposed in the protocol, examine potential unintended effects of policies by examining, in our analysis of specialized managed care for children in foster care in Texas, whether the approach affected those without FDA-approved indications (as intended) without impeding access to those with FDA-approved indications (see “Part 1, Effects of Multimodal Specialized MMCO Intervention”).

Aim 3

Effects of State Monitoring Programs

In this section, we summarize the results for key interventions. We first summarize results from an analysis of the effectiveness of implementation of a multimodal specialized MMCO for children in foster care in Texas. This model used a capitated managed care framework to provide resources, a point of accountability, and the capacity for professional peer review, with a capitation rate designed to reflect the more extensive service needs of children in foster care. It has been an influential model subsequently adopted by other states, highlighting the need for outcomes assessment. Second, we present results from interventions aimed at improving the metabolic monitoring of AP-treated children in Texas and Ohio. Because metabolic monitoring has been a practice on which many states have focused in their AP oversight efforts, this analysis aimed to assess the effects of these interventions to inform future initiatives. Third, we report the results from an analysis of the effectiveness of quality improvement implemented in Ohio county groups, addressing challenges of the implementation of state initiatives in states where child welfare services are locally managed. Fourth, as medical home approaches have been another important model in the landscape of oversight strategies, we present the results from an analysis of the effectiveness of a foster care medical home model on metabolic monitoring in Wisconsin. Fifth, we present the results from an analysis of the effectiveness of a prior authorization program using peer review from clinical experts in Washington State. This analysis aimed to address the effects of prior authorization strategies, another model of considerable importance in the national landscape of psychotropic oversight programs.

Part 1, Effects of Multimodal Specialized MMCO Intervention in Texas

Results from the analyses under this PCORI-funded project have been published in the Journal of the American Academy of Child and Adolescent Psychiatry.¹⁴ In this section, we summarize results as reported in greater detail in this published article. In these analyses, we examined outcomes associated with implementation of a multimodal program that incorporated drug use review, centralized health records, psychiatric consultation lines, and other strategies in an MMCO model in Texas. In the analysis of outcomes, we stratified the patient population into children with an FDA-approved indication, those with other externalizing conditions, and those with other internalizing conditions. Consistent with stakeholder concerns regarding the potential for monitoring programs to limit access to AP medications when they were clinically indicated, our study examined whether the specialized MMCO affected those without FDA-approved indications (as intended) without impeding access to those with FDA-approved indications. As discussed further here, we found that the reduction in AP prescribing was concentrated among children without FDA-approved indications, without apparent adverse effect on indication-consistent treatment. Thus, this stratification addressed an important issue concerning potential unintended consequences of oversight initiatives. Before the development of the specialized MMCO initiative, Texas had one of the highest rates of AP prescribing in the nation among youth in foster care.⁶ During the mid-2000s, this pattern became the focus of considerable public, press, and state legislative concern. The implementation of the specialized managed care program reflected an understanding by key policy makers that there was a need for improvement in access to nonpharmacological alternatives; state guidance as to best practices, and, conversely, practices calling for additional review of a child's clinical status (ie, independent peer review and discussion with the prescribing clinician); and the creation of the capacity for expert peer review. Supported by explicit state prescribing parameters used by the MMCO to guide providers, the program was implemented by Texas in 2008.

In this component of our PCORI-funded study, we examined changes in AP prescribing following the implementation of the MMCO program. The specialized managed care strategy was designed to provide a structure for psychiatric medication reviews with the goal of more judicious prescribing, and support for the higher level of services typically needed for this population, through an enhanced per-member-per-month reimbursement substantially higher than that paid for the general population of Medicaid-insured children. These key structural elements—a specialized managed care structure for the foster care population, explicit state prescribing parameters, and the capacity for retrospective, collegial peer review involving child and adolescent psychiatrists at the managed care plan or state child welfare agency—were highly innovative at the time of implementation and have been adopted or considered for adoption by many states. Hence, examining the effects of the implementation of this intervention was important. Adopted youth were selected as the comparison population to control for secular trends affecting children not subject to or eligible for the MMCO intervention; work under aim 1 established that by the selected post-period, virtually all children in foster care had been transitioned to the new program.

The study population for this analysis comprised children in Texas aged 6 to 17 years in foster care and children not in foster care who were eligible for Medicaid by virtue of their status as “adopted.” Youth in foster care and adopted youth were identified from annual state-specific eligibility codes in MAX personal summary files. This resulted in 102 997 observations (person-years) over 4 years. Because the outcomes are operationalized as annual health service use summaries derived from billing data reported over the entire year, inclusion in the analysis required enrollment in the Medicaid program throughout any calendar year (N = 87 817). We excluded youth whose Medicaid eligibility category was being blind, disabled, or medically needy (because of clinical complexity) or those with dual status (because Medicare is the first payer for prescription drugs for this population, causing Medicaid pharmacy data to potentially be incomplete), resulting in 86 504 observations. Youth without an indicated mental health diagnosis on any pharmacy claims were removed from the sample; this excluded between 0.84% and 2.85% of the sample for any of the study outcomes.

Figure 2 describes the inclusion criteria and exclusions for the study sample. The design was a DID analysis that compared change before and after implementation of the customized MMCO for children subject to the intervention (children in foster care) with a group of children not subject to the intervention (adopted children, fee-for-service payment model). Three outcome variables were studied: (1) use of any AP, (2) use of AP for more than 90 days, and (3) receipt of glucose and lipid monitoring. Logistic regression was used to implement the DID estimation comparing change from the pre-period (2006-2007) to the post-period (2009-2010). Exposure to the intervention was indicated by an indicator for foster care vs adopted status. Models included a pre-period (2006-2007) vs post-period (2009-2010) indicator along with demographic and interaction terms (interaction of foster care and post-period indicators, sex, race/ethnicity, and age). As an enrollee may contribute to the analytical sample for more than 1 observation, cluster-robust standard errors were estimated, allowing for intragroup correlation. From these models, we calculated the population average of predicted probabilities by foster care status and period, using the Δ method to calculate their standard errors.³¹ The outcome measure predicted probabilities (O) were used to make inferences for the linear combination representing the DID parameter = (O_{foster, post} − O_{foster, pre}) − (O_{adopted, post} − O_{adopted, pre}).

Figure 2

CONSORT Diagram for Inclusion and Exclusion Criteria.

To investigate whether differential effects occurred after exposure to the MMCO intervention based on the race or ethnicity of the youth, we included terms to examine the interaction between exposure and race/ethnicity. These results are available in online supplementary tables with the published article by Mackie et al.¹⁴

Within the subgroup of youth with FDA-indicated conditions, the adjusted predicted probability that a youth received any AP medication dispensed before implementation of the multimodal MMCO initiative (years 2006-2007) was 83.0% among youth in foster care and 71.9% among adopted youth (Figure 3). Among youth with FDA-indicated conditions, changes (pre-MMCO vs post-MMCO) among youth in foster care did not differ statistically from those for adopted youth, with a modest increase in both groups. Within the subgroup of youth with other externalizing conditions and no FDA-indicated conditions, the predicted probability of AP dispensing was 44.0% among youth in foster care and 23.4% for adopted youth before implementation (2006-2007). Following implementation (2009-2010), youth in foster care with other externalizing conditions, often referred to as disruptive behavior disorders (attention-deficit/hyperactivity disorder, oppositional defiant disorder, and conduct disorder), experienced a reduction of 6.3 percentage points relative to adopted youth in any AP medications being dispensed (P < .05). Within the subgroup with only other internalizing conditions, the predicted probability of any AP dispensing was 12.0% among youth in foster care and 11.3% among adopted youth before implementation (2006-2007). Youth in foster care with other internalizing conditions (eg, depressive disorders or anxiety disorders) experienced a reduction of 7.6% relative to adopted youth in any AP dispensing in the 2 years following implementation of the multimodal MMCO intervention (P < .05). To summarize, MMCO implementation significantly reduced AP medications prescribed to youths without FDA-indicated conditions while not significantly affecting AP medications prescribed to youths with FDA-indicated conditions.

Figure 3

Change in Average Predicted Probabilities of Any AP Medication Dispensing After Implementation of an MMCO for Youth in Foster Care in Texas.

Within the subgroup of youth with FDA-indicated conditions, before the implementation of the initiative, the predicted probability of AP dispensing over 90 consecutive days was 69.2% among youth in foster care. Among youth with FDA-indicated conditions, changes among youth in foster care did not differ statistically from those for adopted youth, despite a modest increase among youth in foster care (1.5%) and a modest decline (−2.5%) among adopted children. Among youth in foster care with other externalizing disorders, the predicted probability of 90 days of consecutive dispensing was 32.4% before implementation (2006-2007). This rate declined by 4.7% after implementation of the multimodal MMCO intervention. The decline was significantly different from that among adopted youth, who experienced a small (and statistically insignificant) increase in the outcome. After implementation of the Texas policy, enrolled youth with other externalizing conditions experienced a significant reduction in dispensing over 90 consecutive days of −5.5% (P < 0.05) relative to adopted youth. Before the implementation (2006-2007), among youth with other internalizing disorders, the predicted probability of dispensing over 90 consecutive days was 8.0% for youth in foster care, and 5.8% for adopted youth. After implementation, youth in foster care with other internalizing disorders experienced a decline (−4.5%), while the comparison group experienced a small (and statistically significant) increase. Enrolled youth in foster care with other internalizing disorders experienced a reduction in dispensing over 90 consecutive days of −5.1% (P < .05) relative to adopted youth.

Part 2, Metabolic Monitoring of AP-Treated Children in Texas and Ohio

Blood testing to monitor the effects of AP medications on blood glucose and lipids is strongly recommended by clinical guidelines, but implementation of this practice has fallen far short of optimal in most states. To address the effectiveness of state oversight practices and policies for this recommendation, we conducted 2 substudies. The first analysis (see “Part 1, Effects of Multimodal Specialized MMCO Intervention in Texas”) examined change in the rates of glucose and lipid testing for children in foster care in Texas compared with adopted youth, following the implementation of the specialized MMCO program for children and youth in foster care. These analyses were stratified on the presence of clinical conditions with FDA indications for AP treatment, other externalizing conditions without FDA indications, and other internalizing conditions, and were included in the above-referenced paper in Journal of the American Academy of Child and Adolescent Psychiatry.¹⁴ As noted, we found that rates of monitoring were very low in 2006-2007, ranging among youth in foster care from 12.9% to 29.0% across the clinical subgroups and from 9.8% to 19.0% among adopted youth (Figure 4). Following implementation of the MMCO program, testing improved for all groups, and rates of increase for youth in foster care were similar statistically to those for adopted youth. The predicted probability of glucose and lipid testing after intervention exposure ranged from 36.4% (youth in foster care with FDA-indicated conditions, shown in red in Figure 4) to 12.4% (adopted youth with other internalizing conditions, shown in green in Figure 4). Despite the increases, only a minority of AP-treated children in foster care were receiving monitoring by 2009-2010.

Figure 4

Change in Average Predicted Probability of Metabolic Monitoring After Implementation of an MMCO for Youth in Foster Care in Texas.

In the second analysis, using more recent data submitted by the state, we conducted a DID analysis of change in metabolic monitoring from 2010 through 2017 for 2 states, Texas and Ohio, comparing children in foster care in 1 state directly with those in another state. Texas implemented a policy change specifically tailored to metabolic monitoring for children in foster care, while Ohio did not. The intervention studied was the inclusion of a requirement for metabolic monitoring in specific state medication treatment guidelines (psychotropic medication use parameters) in Texas, implemented in September 2013.

Before 2013, Texas had not undertaken major initiatives targeted specifically at improving metabolic monitoring for AP-treated children. In 2013, however, a requested external review of the prescribing guidelines, followed by advice from a within-state advisory group convened by the state, recommended the inclusion of metabolic monitoring as one of the key “practices indicating need for further review of a child's clinical status”¹⁶ in the guidelines; this was added in September 2013 to the 8 existing so-called “red-flag criteria” that triggered a review. This policy change was identified through work under aim 1 as an important mechanism that was amenable to analysis with DID methods, as it took place at a discrete time point and was specifically tailored to the practice in question. The comparison state, Ohio, did not implement interventions specifically addressing metabolic monitoring (the state's primary initiative on AP treatment, the MM initiative, focused on polypharmacy and on AP prescribing for children older than 6 years of age).

In Texas, specific state guidelines (psychotropic medication use parameters) inform the use of psychotropic medication for children in foster care. A central feature of the parameters is a list of “criteria indicating need for further review of a child's clinical status,”¹⁶ which guides the selection of cases for clinical review of a patient's care and helps guide clinicians in identifying practices of concern. Until 2013, the criteria did not address metabolic monitoring. However, in September 2013, after review and input from experts within and outside the state,³² a revised version of the parameters added an additional, ninth criterion to the list: “antipsychotic medication(s) prescribed continuously without appropriate monitoring of glucose and lipids at least every 6 months.”¹⁶ We examined changes in the prevalence of metabolic monitoring in Texas following implementation of the revised guidelines. Because new evidence on the metabolic risks of APs in children was becoming available during this period, along with other changes, such as the publication of treatment recommendations for management of maladaptive aggression in youth and the development of the HEDIS metabolic metric,³³ changes could reflect non–state-specific factors influencing secular trends in monitoring practices. To control for this possibility, we incorporated a comparison with Ohio, which did not implement new policies targeted to metabolic monitoring proximal to this time point.

Similar to the HEDIS measure of “metabolic monitoring for children and adolescents on antipsychotics,”³³ the measure of metabolic monitoring used a 12-month time window to examine the proportion of children and youth aged 0 to 17 years with AP prescriptions during the window who received both a blood glucose and a lipid (low-density lipoprotein cholesterol [LDL-C]) test during that period. For purposes of graphic display of trends, which was desired by our SAB and state partners, we calculated the measure at quarterly intervals. Because the measure assesses the receipt of these tests over a 12-month period, consistent with HEDIS measures that have become industry standard and are widely used by states and health plans, the quarterly measures reflect a “smoothed” trend with overlap of the observation periods. We calculated the measure at quarterly intervals with a 1-year lookback, providing a set of 25 sequential data points in which consecutive periods overlap 9 shared months, resulting in a smoothed curve with the necessary 1-year observation period. Children and youth were eligible for the measure if they were eligible for Medicaid for 11 months of the observation period. Enrollees were considered to be in foster care if they had any period of foster care during the 12-month observation period, based on Medicaid eligibility codes for foster care–based Medicaid eligibility. We defined the preimplementation period as July 2010-July 2013 and the postimplementation period as July 2014-July 2017. We examined the level (intercept) and trend (slope) of the quarterly metabolic monitoring measure during the pre-period and post-period in Texas (state A) and Ohio (state B). The results are shown in Figure 5. During the pre-period, the level of monitoring was consistently lower in Texas than in Ohio, with a similar, gradually increasing upward trend. In the post-period, both the level and trend in monitoring were higher in Texas than in Ohio. The results from the DID analysis are shown in Table 6. In Texas, the mean monitoring rate increased from 32.5% in the pre-period to 46.2% in the post-period, while increasing from 39.2% to 42.2% in Ohio (DID, 10.7 percentage points). The trend (slope) increased by 0.44% per quarter from the pre-period in Texas while decreasing by 0.25% per quarter in Ohio (DID, 0.69% per quarter; Table 7). By mid-2017, more than half of children in foster care in Texas were receiving both glucose and lipid testing (almost 70% received glucose testing).

Figure 5

Level and Slope of Metabolic Monitoring Rates in Texas (State A) and Ohio (State B) Before and After the Addition of Metabolic Monitoring to Texas Prescribing Parameters.

Table 6

DID in Texas vs Ohio Mean Monitoring Rates, Percentage With Monitoring, Pre- and Post-Periods.

Table 7

Rate of Increase per Quarter, Percentage With Monitoring, Pre- and Post-Period.

The results indicate that after the addition of metabolic monitoring of AP-treated children to statewide psychotropic prescribing parameters, metabolic monitoring rates increased substantially among children and youth in foster care, compared with secular trends, as reflected in data on children and youth in foster care in a state that did not implement new interventions or policies focused specifically on metabolic monitoring.

Part 3, Effectiveness of Quality Improvement Implemented in Ohio County Groups

Our third analysis examines the Ohio MM Initiative, which employed an innovative approach in relying on multiple modalities (online modules, fact sheets, and shared-decision-making tools; routine performance benchmarks; and academic detailing) to support clinicians and other stakeholders in improving safe and judicious AP medication prescribing for children in foster care. The multimodal and learning collaborative approach to improve AP prescribing has had only limited study to date but demonstrates promise.³⁴ Given the state initiative's stated goal and emphasis on polypharmacy, prompted by relatively high prior rates of this practice, we examined trends and within-state variation in APC, defined as concurrent filled prescriptions for 2 or more AP medications, before and after the implementation of MM, comparing patterns in the MM counties and non-MM counties. Because resource constraints limited the ability for full initial implementation statewide, the state created three 4-county pilot communities across the state that received technical assistance from a panel of experts in implementing the MM tools in their communities.³⁵ The MM tool kits and educational materials were made available to other counties on a state website, but these counties did not receive the same level of support for practice-based implementation of the tools, and the intervention counties were expected to demonstrate the greatest impact of the program.

To examine change in intervention and nonintervention counties, we therefore grouped the state's counties by intervention status, comparing the 12 MM counties with the other counties in the state. Grouping the counties this way provided adequate cell size to avoid confidentiality problems and adequate statistical power for within-state comparisons, which initial analyses (contrary to expectations) found were not available for individual county estimates, in part because of the overall reduction in AP medication prescribing rates that produced smaller denominators than projected for the quality measures, which are calculated only for AP-treated children. However, grouping MM and non-MM counties provided 2 adequately powered comparison groups for an informative within-state analysis that allowed us to examine whether the statewide effects of MM were associated with the components of the MM initiative that took place only in the pilot counties (eg, practice-based piloting of tools) or with the overall statewide promulgation of the MM materials, educational efforts, and goals.

We examined APC (concurrent filled prescriptions for ≥2 different AP medications) before and after implementation of MM and compared MM counties with non-MM counties. We looked first at effects for all Medicaid-insured children (foster care and non–foster care). In MM counties, APC among all children (foster care and non–foster care) receiving Medicaid benefits decreased from 7.1% in the first period (ending December 2010) to 3.6% in the final period (ending June 2018), representing a relative decrease of 49.3% during that time span. The trend in non-MM counties was similar. In the first period, 6.7% of all children experienced APC, but this number dropped to 3.2% by June 2018, representing a 52.2% decrease (see Figure 6). MM and non-MM counties experienced similar reductions in APC.

Figure 6

Use of APC in Children and Adolescents by County and Foster Care Status, Aged 1 to 17 Years in 2010-2018, Ohio.

We then looked separately at effects by foster care status. Those in foster care had higher rates of APC than did other Medicaid-insured children during the entire study period, though the decline in the percentage receiving APC was steeper for children in foster care than was the decline in the non–foster care group (see Figure 7). In the fourth quarter of 2010, 11.7% of children in foster care and 6.5% of non–foster care children received APC. During the second quarter (Q2) of 2018, these figures were 4.3% and 3.1%, respectively. The decrease was 63.2% among children in foster care and 52.3% among non–foster care children (Table 8).

Figure 7

Use of APC in Children and Adolescents by Foster Care Status, Aged 1 to 17 Years in 2010-2018, Ohio.

Table 8

Use of APC by Age Group for Foster Care and Non–Foster Care Counties in Ohio.

Overall, APC use among children aged 1 to 17 years decreased >60% for children in foster care and more than 50% for children not in foster care. Despite a larger decrease among children in foster care, the rates of APC were higher throughout the study period among children in foster care than in those not in foster care, but a pattern of convergence took place, with rates for children in foster care falling almost to the level of non–foster care children by mid-2018 (Table 8). APC use declined similarly across both MM and non-MM counties, as can be seen in Figures 6 and 7.

Rates of APC were extremely low among children aged 1 to 5 years, regardless of foster care status and MM county status. This finding is promising, as one of the stated goals of the MM initiative was to reduce AP use among children younger than 6 years of age.

The MM goal of a 25% reduction in the use of 2 or more concomitant AP medications for longer than 2 months was achieved by July 2014 for a majority of the age groupings of children, and nearly all the groupings that almost reached such reduction in July 2014 did reach the reduction in the following quarter. Our results parallel those from Thackeray and colleagues,³⁴ who also investigated the MM initiative. They found a significant decrease in APC among both a group of high-volume general pediatric practices and a group of high-volume prescribers of psychotropic medications (decreases of 23.8% and 29.8%, respectively; both P < .01).

We found that following the MM initiative, APC rates fell substantially to lower levels that were sustained over time, with the largest changes observed among children in foster care. However, the effects were not limited to the MM counties but instead appear to have influenced practice statewide. A possible explanation for this pattern, informed by the county-level key informants interviews, is that although child welfare programs are administered at the county level in Ohio and many differences exist in county-level approaches to informed consent and related processes, the MM initiative may have contributed to redefining the avoidance of polypharmacy as a best practice on a statewide level. The specific implementation activities in the county consortia may have been less critical than the promulgation of the MM educational materials, which may have helped redefine polypharmacy across the state's provider community and child welfare agencies as a practice to be avoided.

Part 4, Effectiveness of a Foster Care Medical Home Program for Metabolic Monitoring in Wisconsin

In Wisconsin, beginning in January 2014, all children in and entering foster care in 6 counties (Kenosha, Milwaukee, Ozaukee, Racine, Washington, and Waukesha) were assigned to be enrolled into Care4Kids, a medical home practice model, and assigned a health care coordination (HCC) team that provides and facilitates case management and care coordination services guided by trauma-informed principles, a treatment stance that recognizes and addresses the effects of trauma, which is prevalent among children in foster care. The HCC team develops a comprehensive health care plan (CHCP) for each child within 60 days of entering foster care and updates it every 6 months or when indicated by a significant change in the child's health status. If appropriate, the CHCP also includes an action plan for behavioral health management, including a list of progressive interventions to resolve/de-escalate an emotional crisis/safety situation for children with emotional, behavioral, mental health, and substance abuse problems. This initiative was identified as an important model for study, because many of the state's children in foster care reside in this the 6-county region and because care integration is a central issue in achieving optimal and safe management of AP therapy. For example, improving rates of metabolic monitoring typically involves improving coordination between mental health and primary medical care providers.

To examine this issue, we conducted several analyses of trends and patterns of AP use and management before and after the implementation of Care4Kids. While some of these analyses were still under way in May 2019, initial results suggest that metabolic monitoring rates among youth in foster care increased in the counties implementing the program relative to those in other Wisconsin counties. We conducted analyses of metabolic monitoring rates among youth in foster care, stratified by residence in the 6-county area where Care4Kids was implemented vs residence in the rest of the state (see Figure 8).

Figure 8

Metabolic Monitoring for Children and Adolescents Taking APs by Foster Care Status: All Ages, December 2009 to June 2017, Wisconsin.

Before the Care4Kids program implementation, metabolic monitoring rates were similar in the 6 intervention counties and in the rest of the state, following a parallel trend. Following the program implementation, metabolic monitoring rates increased from 37.4% to 49.1% from the 12 months ending December 2013 to the 12 months ending December 2014, a change of 11.7 percentage points, while remaining relatively stable in other counties in the state (increase of 1.2 percentage points; Table 9).

Table 9

Wisconsin Metabolic Monitoring, Care4Kids Counties vs Other Counties, for Children in Foster Care.

Part 5, Effectiveness of Peer Review Prior Authorization on AP Prescriptions in Washington

This analysis focused on the effectiveness of a Washington State mandatory peer review program on AP prescription rates, accounting for secular trends that affected AP prescribing nationally. The results have been published in Health Services Research³⁶ and are summarized below.

The peer review program was implemented as part of a prior authorization policy in the state. Prior authorization policies, which require that AP prescriptions be approved by the state or its agent (eg, managed care plan) are of considerable importance in the national landscape of AP oversight policies, as a majority of states have now adopted such policies in various forms. The Washington program is of particular interest because it incorporates a 60-day window during which AP treatment may be initiated without prior approval, pending collegial peer review (ie, review by an expert clinician in a process that encourages a professional discussion with the reviewers with the mutual goal of identifying safe and effective strategies for the child's treatment). This is an important model, as the 60-day window incorporates flexibility that could avoid potential problems of undertreatment in time-sensitive situations, although the availability of the time window does create the possibility that some treatments will be initiated that could be avoided through a discussion before treatment is approved. Therefore, it is important to assess the outcomes of this model, which is more flexible than prior authorization models that prevent an initial prescription from being filled until state approval is received. This makes the program something of a hybrid between prospective and retrospective peer review and one whose effectiveness is important to understand.

The program is also distinctive in the collegial nature of the state peer review, which is conducted by University of Washington child psychiatrists with experience in providing voluntary consultations, primarily for primary care physicians managing children with severe emotional disturbances. Trends following implementation of this program have been studied by Barclay et al,³⁷ but their study lacked a control population to measure secular trends during a period of national changes in prescribing. To address this issue, we used an innovative synthetic cohort method to identify an appropriate, multistate comparison population to control for secular trends, with comparisons with other states made possible by use of the MAX data. The synthetic control method uses a data-driven and objective procedure for selecting the comparison group.³⁸ Our application of a synthetic control generates a hypothetical counterfactual region—the comparison region—that uses the Washington data before implementation of the mandatory program to draw a weighted average from the states without monitoring programs during the study period. Data from comparison states were then used to simulate the prevalence of an AP use trajectory that Washington would have witnessed in the absence of the policy and to measure the policy's impact on AP prescribing rates. Prescription claims records from Washington State were compared with a synthetic control group, the members of which were drawn from 19 potential donor states that had not implemented any AP prior authorization program or mandatory peer review for Medicaid-insured children during the period of observation of rates in Washington State. The procedure that generated the synthetic control calculated a weighted average of prescribing rates in 19 states, where the weights are selected to generate the maximum resemblance between Washington and the synthetic control in the preintervention outcome trajectory as well as other state-level variables that can influence prescribing rates. Measures of AP use over time were calculated for each of the states for all Medicaid-insured children aged 1 to 17 years. We included state-level measures indicative of the robustness and quality of the mental health care delivery system, namely, the percentage living in urban areas, the practice of giving prescriber feedback at the state level, and the size of the population served. This procedure controls for selected state-level characteristics and preintervention use trends but not for individual patient-level clinical or demographic factors.

The results shown in Figure 9 (top panel) indicate that before the policy implementation, AP use prevalence closely tracked that of a weighted average of the synthetic control (6.11 per 1000 children/youth in Washington vs 6.18 points in the synthetic control group). “The outcome variable, monthly point prevalence of antipsychotic use, represents the number of children (aged 1 to 17) with an active prescription for at least one antipsychotic medication on the 15th of each month (index date), per 1000 children.”³⁶ Within 2 years after the policy was implemented, the prevalence decreased to 4.24 points per 1000 children/youth in Washington and remained stable in the synthetic control (6.09 points).

Figure 9

AP Use in Washington and Synthetic Control.

To evaluate the significance of this difference, we pose the question of whether this finding may have been entirely driven by chance. To answer this question, we ran falsification tests³⁸ (analyses of the sensitivity of results to alternative comparisons, sometimes referred to as placebo tests) by applying the synthetic control method to each of the 19 other states, shifting Washington to the donor pool, and plotting the gap between outcome prevalence data for each state and its synthetic control. This iterative procedure is akin to the classic framework for permutation inference and provides us with a distribution of estimated gaps for the states where no intervention took place.³⁸ As Figure 9 illustrates, the estimated gap for Washington during the postimplementation period was unusually large relative to the distribution of the gaps for each of the 19 states in the donor pool. This could also be interpreted in the following manner: If one were to relabel the implementation state in the data set at random, the probability of obtaining results of the magnitude of those obtained for Washington would be small, at <.05 (1 out of 20).

The results from this analysis also provide insight into the distinctive effects of the policy for foster care populations vs other Medicaid-insured children (Figure 10). Initially, the impact on prescribing rates was considerably less than for other Medicaid-insured children. Beginning in the second year following the implementation of the peer review process, however, rates of use declined faster in the foster care group than among other Medicaid-insured children, so that the overall change reached convergence between the groups by July 2011. This analysis can be considered somewhat tentative but suggests the possibility that modifying prescribing patterns with this type of intervention may take longer for children in foster care, possibly reflecting the greater clinical complexity of this population.

Figure 10

AP Use by Foster Care Status.

Discussion

Our findings suggest that states' engagement of providers in comparing their practices with standards of best practice and in achieving buy-in to the goals of quality improvement initiatives are key to the success of their initiatives. This can be accomplished through multiple strategies, as long as they engage the provider community. Each of the 4 states conducted significant initiatives toward this aim, using different strategies. The results of the aim 2 interviews suggest that while state oversight was sometimes seen as intrusive, a significant degree of buy-in was common. These results are consistent with the view that the process of normalization, whereby preferred practices are increasingly defined across actors in the decision-making process as best practices, is key to influencing treatment patterns toward guideline consistency.³⁹

The interventions studied varied along a spectrum from strictly voluntary educational initiatives to those with significant mandatory components. The Texas and Washington initiatives we studied combined educational and collegial strategies for provider engagement with peer review strategies; the results suggest that these strategies have promise for sustained uptake of preferred practices. The Texas initiatives also involved reform of the structure and financing of service delivery systems through a specialized managed care program structure, and the Wisconsin initiative involved structural change in service delivery systems through a foster care medical home model; the results were encouraging in both cases. The Ohio results suggest that significant change in specific practices, such as polypharmacy, can also be attained through primarily educational strategies.

In the complex decision-making environment of care for children in foster care, actors in medication decision-making include not only prescribers but multiple individuals involved in care and oversight for these children. Across these constituencies, whose views were elicited in aim 2, perspectives varied, but a common thread across these groups and foster care alumni was the endorsement of a truly team-based approach to decision-making that engages all participants, including the youth themselves, in a well-informed consideration of treatment alternatives. Such a process requires substantial efforts at improved communication and education across all these groups so that the process of consent can be genuinely informed and so that shared decision-making can become a common clinical practice. Despite the many challenges faced by children in foster care, improvement of care quality and outcomes for this population can draw from a considerable reservoir of goodwill and commitment to improved care and outcomes for these children that were apparent across constituencies. Along the same lines, members of the constituencies involved in the care of children in foster care expressed a considerable desire to improve collaboration and communication across decision makers in children's care to improve quality of care and outcomes. The challenge for these efforts has been for the larger systems to facilitate, rather than hinder, the desired communication and coordination among these often-siloed subsystems and to allocate resources and priority to these efforts.

Under the sequential design of this study, metrics for program outcomes were prioritized, incorporating stakeholder input through a systematic prioritization exercise. The work under aim 1 informed the selection of key interventions in each state that were most relevant and appropriate for analysis with observational methods using claims data. This selection took into account multiple factors, including the relevance of the intervention to potential adopters in other states, power considerations, the policy significance of the model being tested, and the evaluability of the intervention through analysis of pre- and postintervention changes compared with relevant comparators. Metrics and comparison groups for each intervention were selected to reflect the most appropriate available comparison populations and the care processes targeted by the intervention.

The results from the claims-based analyses (aim 3) generally indicated that more than 1 structural approach to state psychotropic oversight can be effective in changing practices. We identified several different interventions that were followed by significant improvement in targeted practice. Specialized managed care models, explicit medication use guidelines, educational strategies, health home models, and collegial peer review appear to be effective strategies for improvement. Although the structural features of oversight approaches are important, broader aspects of implementation are likely also significant determinants of success, such as committed leadership, sustained efforts to engage clinicians in a collegial fashion aimed at achieving buy-in to the state initiatives, leveraging social processes of influence to achieve “normalization” of preferred practices, and an appropriate balance between voluntary and mandatory elements so that clinicians are engaged rather than frustrated.

Across the major interventions we assessed, an important factor contributing to impact was the presence of strong efforts by the state to engage clinicians in a collaborative fashion in quality improvement, often using variations of peer review processes, and to define and disseminate best practices. In Texas, where AP treatment rates for children in foster care were among the highest in the nation, a multimodal specialized MMCO dedicated to serving children in foster care was implemented, whose goals included encouraging more judicious AP prescribing, particularly for children without an FDA-approved indication for AP use. Following implementation, AP use declined (relative to a comparison group not subject to the new program) by 6.3 percentage points among children with externalizing mental health conditions without an FDA indication, from a base of 44%, while not changing meaningfully among those with conditions bearing FDA indications (about 14% reduction relative to the base level). A slightly larger reduction was seen among children with internalizing conditions (7.6 percentage points relative to the comparator).

These changes were of a clinically important magnitude. In Washington State, an intervention employing prior authorization with mandatory collegial peer review was followed by substantial reduction in the prescribing rate while remaining stable in comparators. Thus, these very different approaches were both effective. Each state's approach built on the state's distinctive service delivery environment and history; for example, Washington's initiative built on a history of voluntary peer review through a provider help line.

The theme that interventions that are quite structurally different can be effective was also illustrated by results in Texas and Wisconsin for metabolic monitoring. Both states significantly improved metabolic monitoring of AP-treated children through contrasting approaches. The addition of metabolic monitoring to the Texas psychotropic prescribing parameters for children in foster care was followed by an increase in metabolic monitoring rates of about 14 percentage points, while implementation of a foster care health home program in Wisconsin was followed by a comparable increase of about 12 percentage points.

The interpretation of comparative changes across states is complex and requires taking into account the distinctive characteristics of each intervention and the service delivery environment in which it was implemented, illustrating the importance of the mixed-methods approach. By virtue of their large scale and the complicated systems within which they are embedded, state initiatives such as those we assessed are complex, often multimodal, generally adapted to each state's unique services system and policy environment, and often modified during implementation. Valid assessment of interventions of this kind in large state systems requires methods that are tailored to the nature of the problem and consider the unique context and evolution of each intervention and its adaptation to existing systems. Studying interventions at this scale and complexity constitutes an unusually complex research challenge to which the study's mixed-method approach is well suited. This makes the problem less straightforward than that in studies that examine interventions that are more clearly structured and less evolutionary, involve fewer intervention components, and/or are implemented at a more limited number of sites and settings, which can be more readily compared head to head with quantitative methods. Thus, our study may be somewhat atypical in the PCORI portfolio in its breadth and necessarily sequential approach, with the final selection of initiatives and measures informed by initial work under the first 2 aims. However, this was a necessary consequence of the scale, complexity, and multimodal nature of the state oversight systems. Complexity is inherent in the research topic, and studying these large-scale state-level interventions offers the potential to inform the improvement of systems that powerfully affect hundreds of thousands of children across thousands of settings.

By addressing an important knowledge gap, our findings help states make difficult decisions in implementing psychotropic oversight systems for publicly insured children, especially for children in foster care for whom the state has a unique, quasi-parental responsibility. Oversight systems are complex and poorly documented; little evidence is available on the comparative effectiveness of alternative oversight system innovations, their impact on various stakeholders, and the extent to which they are consistent with the values and priorities of those who are affected by them. These systems affect the welfare of hundreds of thousands of vulnerable children nationally and have implications for the large population of non–foster care Medicaid-enrolled children as well. However, they are not developed to address educational, social, or foster care placement outcomes. Rather, they focus on health and mental health outcomes related to the use of AP medications and other psychotropic drugs. Thus, it is vital to understand the choices available to state policy makers and their consequences. Assessing their effects requires a multidimensional approach.

Each state's oversight programs are embedded in a unique institutional and policy environment. Multiple actors are involved in the decision-making process. Well-tested measures of safe and judicious prescribing of AP medications in children that were developed with broad stakeholder involvement have only recently become available; also, the patient population consists of vulnerable children in state care who cannot feasibly be interviewed directly, and policies involve multiple components. To study the impacts of such large and complex systems, we used mixed-methods strategies that combine systematic documentation and analysis of evolving policies and their components, assessment of policy impacts from the perspectives of multiple stakeholders, and the use of large administrative data sets to measure impacts on large affected patient populations. To address these challenges, we analyzed and documented state psychotropic oversight systems used to provide consultation, prospective review, and retrospective review of psychotropic treatment for children in foster care. We documented key components and the implementation process/timetable of the natural experiments resulting from policy changes. We examined the experiences, values, and preferences of patients and other stakeholders regarding psychotropic oversight systems and identified measures of program impact that are important for foster care alumni and other stakeholders. Finally, we examined the effects of different state psychotropic oversight systems on practices identified as important to patients and stakeholders using evidence-based and consensus-driven quality metrics relevant to each intervention and its objectives.

The complexity of the state oversight mechanisms required a sequential process and triangulation across the information collected under the study's 3 aims. Because of the underdocumented nature of the state oversight systems, document reviews and key informant interviews (aim 1) were necessary to inform the selection and prioritization of state interventions for outcomes analysis before specific interventions could be selected for claims-based analyses. Because the outcomes analyses required the use of claims data not developed for research purposes, extensive work was also necessary to assess data completeness and other issues specific to each state's Medicaid data system; such work is key for validity in future studies using these data sources. For the claims-based analysis portion of the project, we took into account factors such as whether interventions took place at a sufficiently clear-cut point in time for pre-post analysis, involved clearly defined policy/administrative changes of significant scope, targeted specific practices identified as important by stakeholders and patients, and had appropriate comparison populations available. Substantial work was done to assess the adequacy of data from each state's Medicaid system for research analysis, with substantial data checking, exploration, and creation of descriptive results as well as checking for any anomalies that might raise issues of data completeness. Descriptive results were fed back to the participating states for review, comment, and comparison to any internal analyses conducted. Although it is too early to assess changes made by states in response to this feedback, states are using it to inform their ongoing program choices and refinement of strategies, and at least 1 state has updated the analyses provided to them using its own data to assess the most recent trends.

We also consulted with the SAB and patient partners. Based on these considerations and on the resource-intensive nature of each analysis, we selected the following for further claims-based pre-post analysis: the implementation of a multimodal specialized MMCO in Texas, the impact of new state guidelines on metabolic monitoring in Wisconsin, the impact of the MM project in Ohio, and the effects of peer review prior authorization on AP prescriptions in Washington. We used the results from aim 1 to identify appropriate comparison groups for each of these analyses. In addition to these focused analyses of specific state interventions amenable to pre-post analysis, we conducted a systematic analysis of patient and stakeholder group priorities concerning outcome measures for interventions that are important to these groups. We used an innovative approach to elicit these priorities and preferences with 3 purposes in mind: (1) to inform the prioritization of outcome measures; (2) to test and demonstrate an approach that may have broader utility across patient-centered comparative outcome studies for identifying outcomes important to patients and stakeholders (“patient-centered outcomes”); and (3) to inform ongoing national and state quality measure development efforts (eg, HEDIS,¹⁰ National Quality Forum,⁴⁰ CMS Core Pediatric Quality Measures for Medicaid,⁴¹ and measures used by individual states and health plans to assess safe and judicious AP prescribing). These core quality measurement suites are reviewed by experts and stakeholders and can be revised annually for relevance and feasibility. The identification of patient and stakeholder priorities, examined in this work, has the potential to provide evidence to inform these highly influential processes.

In addition, the rich qualitative data from aim 2 from alumni, clinicians, caregivers, and prescribers provide important insight into the perspectives of these stakeholder groups, as described in the Results section. The results suggest that these actors often feel that their perspectives are insufficiently understood or addressed by one another. Understanding and addressing these issues are vital steps in helping systems move toward genuine shared decision-making and patient-centered approaches to the management of children who have experienced severe emotional disturbance and trauma leading to mental health and behavioral difficulties. This requires clinicians; professionals involved in care management, oversight, and the informed consent process; and others involved in children's lives to learn to listen more productively to one another and to incorporate youth perspectives in decision-making. The results of the aim 2 work provided important insights that could not have been gained by other methods. Insights were particularly informative on the issue of informed consent and shared decision-making in consent and assent for medication use, an issue of fundamental importance for youth proposed to be treated with powerful medications that involve both opportunity for improved self-regulation and significant risks and adverse effects, some of which look different from the patient's and clinician's perspectives. For example, sedation associated with these medications can be a significant quality-of-life, educational, and social concern for AP-treated children. The question of who provides medication consent emerged as a vital issue for each of the stakeholder groups, with a notable amount of agreement across stakeholder groups that this complex issue requires individualization and that simple, formulaic rules as to who can consent are problematic. Although it might be assumed that there is a fundamental divide between youths' desires to make their own decisions in these matters and views by others involved in the process that clinical expertise and adult maturity should prevail, the responses, as we described, were actually more nuanced and provided much room for common ground.

In this light, communications and systems challenges emerged as barriers acknowledged on all sides as key for true informed consent. Foster care alumni noted multiple barriers to true informed consent, including insufficient time and information provided by the prescriber to allow for true informed consent/assent, as well as decisions made by clinicians with whom no real relationship had been established and by other adults not closely involved with the child's current life and who were not knowledgeable about the child's concerns. This is significant given the variation in both policy and actual practice on informed consent in child welfare systems, which vary widely as to which adults, in which roles, are responsible for giving consent for a child to take an AP. For example, some systems assign this responsibility to biological parents or to current caregivers (eg, foster care providers); others involve judicial review; and yet others include more direct decision-making by child welfare agency staff. Given the wide range of children's circumstances and the degree to which individuals in these different roles are in a position to provide truly informed consent, respondents communicated the need to individualize the informed consent process and for policies to take account of this need rather than being rigidly driven by inflexible legislative, regulatory, or policy directives. These findings support a recommendation for future research that digs further into the medication decision-making process for youth, including those in foster care and those in other settings where oversight and shared decision-making are at issue, and that examines the impact of alternative processes on these decisions and their outcomes.

Concerns expressed by youth about barriers to their participation in genuine informed consent, as described in the Results section, such as decision-making by clinicians with whom they have developed little relationship, are part of broader concerns about the receipt of mental health care that is predominantly pharmacologically centered rather than incorporating a more psychotherapeutically oriented relationship. In the foster care population, trauma and its psychosocial sequelae are highly prevalent, even typical. Mental health needs and behavioral challenges associated with trauma have historically led to high rates of AP treatment. A challenge to health, mental health, and child welfare systems has been to deploy effective systems of trauma-informed psychosocial intervention for these children as the first-line approach and complement to any necessary psychopharmacological treatment. The priority placed on adequate provision of these services by their consumers was reflected in the strong endorsement of measures of psychosocial intervention by foster care alumni.

It was encouraging that the majority of AP-treated children did receive at least some services coded as psychotherapy rather than simply psychiatric evaluation and medication management; among the states that provided recent (2015-2017) data, performance on a first-line psychosocial care metric, part of the HEDIS AP measurement suite, was generally stable over time and in the 75% to 85% range. The proportion of AP-treated children receiving at least some psychosocial service may be even higher if, in some cases, psychosocial services are provided from non-Medicaid sources; however, this is a minimal criterion in that even 1 service encounter qualifies a child for the metric. Renewed efforts are needed to improve not simply the proportion of AP-treated children with any psychosocial treatment but, more importantly, access to evidence-based, high-quality, trauma-informed psychosocial mental health services of sufficient intensity and duration to engage youth in a supportive, longitudinal relationship that helps them build the skills to manage emotional challenges in a constructive fashion, with or without help from medication. Further development of team-based strategies to provide high-quality, trauma-informed psychosocial mental health services may also help better address the challenges that youth and other stakeholders report in achieving genuine shared decision-making and informed consent, because clinician teams with an established supportive relationship with clients may be better situated to engage in this type of decision-making than are those who provide a more limited, psychopharmacology-based model that does not integrate psychotherapy with medication management. It should be noted that the issues encountered by patients (and reported by alumni) in receiving psychotherapeutic services that they view as adequate, and in the adequacy of communication with prescribers regarding medication choices so that they can participate in genuine informed consent and shared decision-making, are distinct from the issues surrounding external peer review of clinicians' patterns of prescribing and medication management. It would be valuable for future studies to examine in greater detail the type, quality, and quantity of psychosocial mental health services provided to children in foster care with mental health conditions and/or psychotropic use, including the extent to which these services are trauma informed, evidence based, and responsive to the specific needs of children in foster care, including their frequent histories of physical and emotional trauma and adverse childhood experiences.

Receiving psychosocial services also does not ensure the level of communication about medication decision-making needed for genuine informed consent, because these services are not necessarily provided by the physicians responsible for AP prescriptions. Foster care alumni often expressed concerns about the limited communication with prescribers about medication choices, often in relatively brief visits. In the prioritization analysis, foster care alumni rated increasing the proportion of children provided with psychotherapy before or just after starting an AP near the top of their priorities, only slightly below (and statistically indistinguishable from) reduction in the number of children prescribed APs overall and among children aged 0 to 5 years. Although guidelines such as the Treatment of Maladaptive Aggression in Youth^42,43 and the AACAP prescribing parameters²⁵ endorse psychotherapy as first-line treatment before or concurrent with AP treatment, it has not been clear to what extent this service is actually desired by the patient population or is available for the provider community. The results of our study suggest that psychotherapy is actively sought by many patients, suggesting that the HEDIS measure of first-line psychosocial treatment for AP-treated youth³⁰ is consistent with patient priorities and supporting the continuation of this measure as a criterion for the performance of Medicaid health plans. These findings also support the consideration of this metric for other nationally accepted measure sets, such as the CMS Core Pediatric Quality Measures for Medicaid.⁴¹ From the perspective of informed consent and shared decision-making, incorporation of psychopharmacological treatment and psychosocial treatment (such as psychotherapy and counseling) in a comprehensive, team-based mental health intervention strategy may facilitate medication decision-making involving a clinician team with a meaningful, longitudinal relationship with the youth, in which 2-way communication about AP and other psychotropic treatments is supported by an ongoing, supportive relationship.

The claims-based analyses of treatment patterns before and after implementation of specific state policy and program initiatives provide important insights into the impact of state choices among oversight strategies. The analysis of prescribing changes following the implementation of the Texas multimodal MMCO strategy suggests that specialized managed care plans for children in foster care offer a structure and framework that can support changes in provider behavior toward more judicious AP prescribing. As discussed previously, we separately examined the impact on children with conditions that do and do not bear an FDA indication for AP treatment, finding that the program was successful in reducing AP prescribing among children without FDA indications while not significantly affecting treatment for those with conditions bearing FDA indications, suggesting a trend toward more judicious prescribing.

The results from this analysis suggest that specialized managed care structures, like the one implemented in Texas, tailored to the specific needs of children in foster care can reduce AP treatment for youth with conditions where concern is greatest (eg, attention-deficit/hyperactivity disorder [ADHD], anxiety, depression) without significantly reducing AP prescribing among children with conditions where evidence for efficacy and favorable benefit/risk ratio is stronger (eg, schizophrenia, bipolar disorder). It is important to note that in this model, the specialized managed care plan was reimbursed at a higher rate than were the “generic” managed care plans serving the general population of Medicaid-insured children. A future research need is to design studies that are well suited to rigorous examination of other potential adverse outcomes, such as loss of foster care placements, incarceration, expulsion from school, or other negative functional or social consequences. Designs suited to such analysis might include linkage of treatment data with data from child welfare, criminal justice, or educational systems.

The differential changes for children with and without conditions bearing FDA indications for treatment suggest that the Texas MMCO program had effects on prescribing that differed by diagnostic group; as hypothesized, effects in reducing AP use were greater in children who did not have a condition with an FDA indication. It should be noted that the MMCO intervention we studied in Texas was not simply the use of managed care as a payment mechanism; rather, this model entailed additional investment in provider support, care coordination, clinical support, and drug use reviews required under the managed care contract. The MMCO model jointly implemented these tools, which our qualitative work suggests worked together to influence AP prescribing and related processes, such as metabolic monitoring for AP-treated children. Because of the diversity of implementation of models of this kind, however, we would recommend that additional studies assess the outcomes of such models in different states where the policy and systems context, as well as specifics of program design and implementation, are different from those in the states we studied. Since the implementation of the Texas model, several other states have implemented specialized managed care models for children in foster care; systematic assessment of these innovations in these states is warranted. The methods used in our current study, such as mixed-methods designs and DID modeling of changes following implementation, would be useful when states set up studies to assess the effect of these policy interventions. Optimally, requirements and resources for such studies would be embedded in legislation or other mandates authorizing the creation of these programs so that outcomes are assessed and fine-tuned as implementation proceeds.

The Texas model provides a substantially higher per-member-per-month payment to the plan than do the managed care plans that serve the general population of Medicaid-insured children, adapted to the distinctive set of service needs required by children in foster care. This situation is distinct from situations in some states involving the mandated shift of children in foster care into the generic managed care programs designed and financed to serve the general population of Medicaid-insured children. Our study did not address this situation; further research is needed on outcomes for children transitioned into these programs, and the results from Texas should not be assumed to apply to such transitions. The results reported on the effect of adding metabolic monitoring to the Texas foster care parameters illustrate how the specialized managed care model for children in foster care provides a structure that facilitates ongoing improvement of care processes that can be tailored to emerging issues. In its initiative to improve metabolic monitoring, the state was able to leverage the presence of an existing structure for collegial peer review and provider education offered through the specialized managed care program as well as the existence of an explicit set of prescribing guidelines like the Texas prescribing parameters designed to guide improvement of practice. As shown in the Results, implementation was followed by a sharp increase in metabolic monitoring, from 32.5% to 46.2%, a 42% relative increase. DID analyses indicate that the improvement was substantially larger than the change in a comparison state, Ohio, that did not implement initiatives specifically targeted to metabolic monitoring during this period. These results suggest that within the context of an oversight structure that incorporates explicit state guidelines and the capacity for peer review of practice quality, adding requirements to guidelines for specific practices (eg, metabolic monitoring for AP-treated youth) can be highly effective in achieving intended practice changes. The improvement in metabolic monitoring among AP-treated children is noteworthy given the difficulties across the nation in completing these tests that result from the typical separation between mental health treatment systems and primary medical care, given that most psychiatric practices do not have onsite capacity for blood draws. Thus, achievement of this type of improvement likely required improved coordination between mental health and primary care providers in attaining improved monitoring for AP-treated children, although we did not specifically study this issue. Future designs for analysis of integrated-care models for this population could incorporate data collection from care sites or other strategies for assessing care integration. It should also be noted that achieving improved rates of metabolic testing—the measure addressed in the HEDIS quality measures and used in many improvement initiatives—does not guarantee that abnormal test results are addressed clinically in an optimal fashion. Clinical data such as those from electronic medical records could be used in future studies to address this issue.

The need to achieve improved performance in metabolic monitoring for children in foster care has received increased attention in recent years as evidence has accrued of metabolic abnormalities and increased diabetes risk associated with pediatric AP treatment. The addition of a metabolic monitoring metric to the HEDIS measure set,³⁰ growing out of work by the MMDLN/Rutgers CERTs and NCINQ initiatives,^33,34 has also brought renewed focus to this practice. However, the slow pace of improvement indicates that measurement alone is not enough. More research is needed on the comparative effectiveness of state and health plan initiatives to improve metabolic monitoring in AP-treated youth, especially those in foster care. However, the effectiveness of the incorporation of metabolic monitoring into the Texas prescribing parameters provides an important proof of concept that this practice can become the norm for children in foster care when supported by explicit state guidelines and a peer review process.

The results from the analysis of Ohio's MM initiative³⁵ illustrate how a very different approach to quality improvement in pediatric AP prescribing can also be effective. This multicomponent effort included time-limited provider education activities, fact sheets, the setting of explicit goals for reduction in polypharmacy and pediatric AP prescribing, and the piloting of shared-decision-making tools and practice-based improvement programs in 13 pilot counties. Ohio's initiative was voluntary and was not focused on required prior authorization and/or peer review processes. Ohio did not promulgate official state guidelines, but the pattern of change, being statewide rather than limited to the counties where the tool kit was formally implemented, suggests that explicit goals set in the initiative may have functioned as implicit guidelines or definitions of best practice. We examined changes in APC, one of the principal foci of the MM initiative, for which a goal of 25% reduction was set by the state. Once the data were in hand, our initial analyses found that because the base rate of AP prescribing was relatively low and declined during the study period, the analyses were powered only to compare patterns across groups of counties rather than in individual counties; in fact, in many cases, reporting rates for individual counties would have created small cells that would raise confidentiality problems. To have sufficient power to compare outcomes for counties in the MM consortia with counties that were not, we therefore grouped counties accordingly.

We hypothesized that change would be greatest in the counties that were part of the 3 MM multicounty consortia, in which provider education efforts were concentrated and provider coalitions were formed. This hypothesis was disconfirmed. Rates of change were very similar between the 2 county groups. However, implementation of MM was followed by a substantial reduction in APC for both county groups, with rates falling by more than half, well exceeding the target. These results suggest that the effects of the program were statewide. The effects of the more intensive practice-based quality improvement activities and use of the shared-decision-making tool kit in the MM pilot counties may have had less effect than did the promulgation of APC avoidance as an officially endorsed best practice on a statewide level. Specific implementation activities in the county consortia may have been less critical than the promulgation of the MM educational materials that may have helped redefine APC across the state's provider community and child welfare agencies as a practice to be avoided.

As discussed in the Results section, we were able to follow rates of APC prescribing through mid-2018 with updated Medicaid files provided by the state, finding a considerably sharper decline in polypharmacy among children in foster care than among other Medicaid-insured children following MM implementation during the more recent period. In this period, polypharmacy declined faster for children in foster care than for other Medicaid-insured children, resulting in a marked narrowing of the differences that had existed earlier (moving from rates that were almost twice as high among children in foster care to near convergence, with a noteworthy reduction of >60% among children in foster care). Although it is difficult to identify the causes of these apparently different sequelae of an intervention that was not specifically targeted to children in foster care, it is noteworthy that such a substantial reduction was achieved for youth in foster care. A possible interpretation is that the official definition of polypharmacy as a practice to be avoided may have had greater impact among children in foster care, given the higher level of governmental responsibility for and scrutiny of the care of children placed in public care. More broadly, the Ohio experience suggests that setting goals and defining best practices at the state level may have considerable impact on some practices, even without a peer review apparatus supporting it.

The sharp reductions in APC in Ohio, both in foster care and non–foster care populations, contrast sharply with data reported by NCQA for Medicaid health maintenance organizations nationally for the period of 2015-2017, which indicate that the national rate of APC increased on a relative basis by 20%, from 2.5% to 3%, during this period.⁴³ This comparison should be considered somewhat tentative but tends to support the effectiveness of Ohio's efforts to highlight the need for providers to minimize APC.

In Wisconsin, the major intervention was a specialized primary care–based health home model for children in foster care. Implemented only in a 6-county area, this program was not available to children in foster care in the rest of the state. Because of the model's focus on the improvement of primary medical care and its integration with mental health care for this population, metabolic monitoring of AP-treated children was selected as the most appropriate metric for outcome assessment. Metabolic monitoring rates among youth in foster care increased in the counties implementing the program relative to those in other Wisconsin counties.

Finally, analysis of data from Washington State examined the effects of an important variant of the increasingly prevalent prior authorization requirements imposed by states for AP use in publicly insured systems. The results from this analysis suggest that a less stringent prior authorization process that allows an initial 60 days of treatment pending peer review was nevertheless significantly effective in encouraging more conservative prescribing, relative to a national comparator drawn from states that did not implement prior authorization processes during the period of analysis. The generalizability of this success for simple prior authorization is questionable because it occurred in a setting in which the peer review system had been well established. It was implemented in the context of a sustained process of engagement of the state's clinical community in a variety of consultative and educational initiatives, including a well-developed voluntary peer review consultation line, conducted by expert child psychiatrists who ultimately staffed the required peer reviews. Information from the state case study suggests that this process of engagement with the clinical community was key in achieving buy-in from that community and to the success of the required peer reviews. The results from the Washington analysis suggest that, as in Texas, mandated peer review processes can be effective in achieving more conservative AP prescribing for children in foster care. It is important to note that all 4 of the states employed multiple strategies to improve psychotropic—and, particularly, AP—oversight, and that all 4 achieved significant reductions in AP prescribing as well as on quality measures. The results indicate that multiple strategies can be effective in improving prescribing patterns. The major innovations implemented by the states were extremely different in their aims and structure as well as in the aspects of AP use that were targeted (albeit with some overlap), the populations targeted, and other characteristics; therefore, a comparison of state strategies is most appropriate at the broader level of assessing the types of strategies that can be effective, rather than an effort to make a highly quantitative head-to-head comparison between such different interventions. Methodologically, such comparisons are best rooted in analysis of outcomes of each intervention using metrics, time frames, and comparison populations best suited to the aims and nature of each intervention. Nevertheless, many comparisons at a broader level are useful.

Among the dimensions of AP management, metabolic monitoring of AP-treated children was most relevant to interventions in Texas and Wisconsin. The effect sizes associated with these interventions are not directly comparable because each analysis used different time frames and comparator populations tailored to the different state systems and interventions; however, each of the interventions was followed by clinically substantial improvement. We interpret the results as suggesting that improved metabolic monitoring of AP-treated children is achievable in the general population of children in foster care through at least 3 different strategies: (1) creation of a dedicated managed care organization to serve children in foster care, with reimbursement and contracting requirements tailored to this population's needs, as in the Texas MMCO program; (2) explicit state guidelines requiring monitoring, supported by quality measures and peer contacts with providers, as in the 2013 addition of metabolic monitoring to the Texas psychotropic prescribing parameters for foster care; and (3) structural interventions aimed at improving primary care and better integrating it with mental health care, as in the Wisconsin Care4Kids health home program. In practice, these 3 strategies did not take place in isolation from one another. For example, the specialized managed care program that Texas implemented in 2008 brought additional resources to the overall medical and mental health care of children in foster care and created a framework (with guidelines, metrics, and a peer review process) that was subsequently leveraged by other quality initiatives, such as the 2013 addition of metabolic monitoring to the guidelines. Based on all the Texas information from this mixed-methods study, our assessment is that the metabolic monitoring improvements after 2013 are best attributed not simply to the promulgation of state recommendations for this practice but also to the implementation of such a guideline within the structure created by the MMCO and the state's continuing, persistent commitment to monitoring, quality measurement, peer review, and provider education. Our sense from the overall study results is that this type of cumulative building of interventions on one another is common in successful state initiatives to improve psychotropic oversight. Indeed, sustained state commitment to oversight, with periodic new initiatives that continue to focus providers' attention on the quality of care for this vulnerable population, may be 1 of the most important “active ingredients” of effective oversight programs.

It should be noted that the 4 states were not selected randomly but rather are states that were interested in improving their systems and that volunteered to participate. Each implemented significant initiatives during this time period, albeit with differing aims and targets; none can be considered an inactive comparator, and this group of states cannot be considered representative of other states. Significant improvement was observed in targeted practices in each of the 4 states, suggesting that there is not a singular effective strategy but rather that improvement can be achieved through multiple pathways. Common across the state initiatives, however, was the presence of state champions for safer and more judicious AP use and significant efforts in each of the states for provider education and collegial engagement with providers aimed at achieving buy-in to the state initiatives. Information from the 3 aims of the study, taken together, suggests that these are likely common features of successful state interventions to improve psychotropic prescribing and management.

Conclusions

Our results indicate that structural strategies that can be effective in improving AP prescribing for children in foster care include peer review, specialized managed care models, health home models, educational strategies, and specific state guidelines/expectations for best practice. In Texas, several of these elements were incorporated into a multimodal MMCO program providing augmented clinical resources; a peer review process for psychotropic medication use reviews; and provider education capacity, supported by state prescribing parameters. Implementation was followed by reduced AP prescribing for children with weaker FDA indications for treatment. This framework was also effective in achieving improved performance on metabolic monitoring when monitoring requirements were added to the parameters in 2013. Ohio's multimodal MM intervention defined state goals for change in targeted practices and developed practice-based improvement tools implemented in 3 county consortia and made available as resources statewide. Substantial improvement in targeted practices took place both in the consortium counties and in other counties, which may reflect the statewide impact of defining targeted practices as problematic and/or the spread of changed practices across counties.

In Wisconsin, implementation of a foster care–specific medical home model in a 6-county area was followed by improved metabolic monitoring, while monitoring rates declined elsewhere in the state. The results suggest that specialized health home models show promise for quality improvement via improved integration of clinical services. In Washington, mandatory prior authorization with clinician peer review was followed by substantial prescribing reductions, indicating that peer review and consultation by expert clinicians can be effective in lowering AP prescribing rates. Prior implementation of a voluntary provider consultation program contributed to provider buy-in; a 60-day window for treatment without state approval pending peer review did not appear to negate effectiveness.

Across states, recurrent facilitators of success included codification of best practices reinforced by clinically oriented, collegial review with prescribers. These features contributed to provider buy-in and “normalization” of preferred practices, which the qualitative components of the study suggest were important mediators of success. The use of multimodal approaches engaging multiple actors in the medication use process may strengthen buy-in and the incorporation of preferred practices in routine clinical processes. Overall, the results indicate that statewide oversight initiatives are often complex, multimodal, and embedded in system contexts that are very different across states and even localities. Mixed-methods approaches are key for understanding implementation in practice and providing multiple consumer and stakeholder perspectives on their impact. Despite the challenges involved in conducting a rigorous evaluation of such initiatives, convergent evidence from qualitative and quantitative components of this project provided important insights that can inform further program refinement and the adoption of such initiatives across states.

References

1.

Crystal S, Olfson M, Huang C, Pincus H, Gerhard T. Broadened use of atypical antipsychotics: safety, effectiveness, and policy challenges. Health Aff (Millwood). 2009;28(5):w770-w781. [PMC free article: PMC2896705] [PubMed: 19622537]

2.

Medicaid Medical Directors Learning Network (MMDLN)/Rutgers Center for Education and Research on Mental Health Therapeutics (CERTs). Antipsychotic Medication Use in Medicaid Children and Adolescents: Report and Resource Guide from a 16-State Study. Published June 2010. Accessed February 12, 2021. http://nrc4tribes​.org​/files/Antipsychotic​%20Use%20in%20Medicaid​%20Children%20Report​%20and%20Resource%20Guide.pdf

3.

US Government Accountability Office. Report to Congressional Requesters: Children's Mental Health: Concerns Remain About Appropriate Services for Children in Medicaid and Foster Care. GAO-13-15. Published December 2012. Accessed February 12, 2021. https://www​.gao.gov/assets/660/650716​.pdf

4.

US Government Accountability Office. Foster Care: State Practices for Assessing Health Needs, Facilitating Service Delivery, and Monitoring Children's Care. GAO-09-26. Published February 2009. Accessed February 12, 2021. https://www​.gao.gov/assets/290/285993​.pdf

5.

US Government Accountability Office/Kutz GD. Foster Children: HHS Guidance Could Help States Improve Oversight of Psychotropic Prescriptions. GAO-12-270T. US Government Accountability Office. Published December 1, 2011. Accessed February 12, 2021. https://www.gao.gov/assets/590/586570.pdf

6.

Rubin D, Matone M, Huang Y-S, dosReis S, Feudtner C, Localio R. Interstate variation in trends of psychotropic medication use among Medicaid-enrolled children in foster care. Children Youth Serv Rev. 2012;34(8):1492-1499.

7.

Office of the Inspector General. Second-Generation Antipsychotic Drug Use Among Medicaid-Enrolled Children: Quality-of-Care Concerns. Report OEI-07-12-00320. US Department of Health and Human Services. Published March 25, 2015. Accessed February 12, 2021. https://oig​.hhs.gov/oei​/reports/oei-07-12-00320.pdf

8.

Crystal S, Mackie T, Fenton MC, et al. Rapid growth of antipsychotic prescriptions for children who are publicly insured has ceased, but concerns remain. Health Aff (Millwood). 2016;35(6):974-982. [PubMed: 27269012]

9.

Mackie TI, Kovacs KM, Simmel C, Crystal S, Neese-Todd S, Akincigil A. A best-worst scaling experiment to identify patient-centered claims-based outcomes for evaluation of pediatric antipsychotic monitoring programs. Health Serv Res. 2021;56(3):418-431. doi:10.1111/1475-6773.13610 [PMC free article: PMC8143685] [PubMed: 33369739] [CrossRef]

10.

HEDIS measures and technical resources. National Committee for Quality Assurance. Accessed May 31, 2019. https://www​.ncqa.org/hedis/measures/

11.

Srivastava A, Thomson S. Framework analysis: a qualitative methodology for applied policy research. J Admin Govern. 2009;4(2):72-79.

12.

Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):117. doi: 10.1186/1471-2288-13-117 [PMC free article: PMC3848812] [PubMed: 24047204] [CrossRef]

13.

Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277-1288. [PubMed: 16204405]

14.

Mackie T, Cook S, Crystal S, Olfson M, Akincigil A. Antipsychotic use among youth in foster care enrolled in a specialized managed care organization intervention. J Am Acad Child Adolesc Psychiatry. 2020;59(1):166-176. [PubMed: 31071384]

15.

Texas Department of Family Protective Services. Child Adolescent Needs and Strengths (CANS) assessment. Accessed June 1, 2020. https://www​.dfps.state​.tx.us/Child_Protection​/Medical_Services/CANS_Assessment​.asp

16.

Texas Department of Family and Protective Services, The University of Texas at Austin College of Pharmacy, Federation of Texas Psychiatry, Texas Academy of Family Physicians, Texas Osteopathic Medical Association, Texas Medical Association. Psychotropic Medication Utilization Parameters for Foster Children. Published January 2007. Accessed May 31, 2019. http://www​.dfps.state​.tx.us/child_protection​/medical_services​/documents/reports/2016-03​_psychotropic​_medication_utilization​_parameters_for_foster_children.pdf

17.

dosReis S, Tai MH, Camelo WC, Reeves G. A national survey of state Medicaid psychotropic-monitoring programs targeting youths. Psychiatr Serv. 2016;67(10):1146-1148. [PubMed: 27133725]

18.

Mackie TI, Hyde J, Palinkas LA, Niemi E, Leslie LK. Fostering psychotropic medication oversight for children in foster care: a national examination of states' monitoring mechanisms. Adm Policy Ment Health. 2017;44(2):243-257. [PubMed: 26860953]

19.

Mackie TI, Hyde J, Rodday AM, et al. Psychotropic medication oversight for youth in foster care: a national perspective on state child welfare policy and practice guidelines. Child Youth Serv Rev. 2011;33(11):2213-2220.

20.

Dimick JB, Ryan AM. Methods for evaluating changes in health care policy: the difference-in-differences approach. JAMA. 2014;312(22):2401-2402. [PubMed: 25490331]

21.

Ruttner RB, Nysenbaum J, Williams S. Guide to MAX data: Medicaid policy brief 2015. Mathematica Policy Research; 2015. Accessed February 12, 2021. https://www​.cms.gov/Research-Statistics-Data-and-Systems​/Computer-Data-and-Systems​/MedicaidDataSourcesGenInfo​/Downloads/MAX​_IB21_MAX_Data_Guide.pdf

22.

Lockwood KK, Friedman S, Christian CW. Permanency and the foster care system. Curr Probl Pediatr Adolesc Health Care. 2015;45(10):306-315. [PubMed: 26403649]

23.

Courtney ME, Dworsky A. Early outcomes for young adults transitioning from out-of-home care in the USA. Child Fam Soc Work. 2006;11(3):209-219.

24.

Cusick GR, Courtney ME. Offending During Late Adolescence: How Do Youth Aging Out of Care Compare With Their Peers? Chapin Hall Center for Children at the University of Chicago; 2007.

25.

American Academy of Child and Adolescent Psychiatry. Pharmaceutical benefit management and the use of psychotropic medication for children and adolescent. Published January 12, 2009. Updated January 2014. Accessed May 31, 2019. https://www​.aacap.org​//aacap/Policy_Statements​/2009/Pharmaceutical​_Benefit_Management​_and_the_Use_of​_Psychotropic_Medication​_for_Children_and_Adolescent.aspx

26.

Loy JH, Merry SN, Hetrick SE, Stasiak K. Atypical antipsychotics for disruptive behaviour disorders in children and youths. Cochrane Database Syst Rev. 2012;(9):CD008559. doi: 10.1002/14651858.CD008559.pub2 [PubMed: 22972123] [CrossRef]

27.

Correll CU. Monitoring and management of antipsychotic-related metabolic and endocrine adverse events in pediatric patients. Int Rev Psychiatry. 2008;20(2):195-201. [PubMed: 18386212]

28.

Ohio Medicaid. Ohio Minds Matter. Accessed May 31, 2019. http:​//ohiomindsmatter.org/

29.

PCSAO Behavioral Health Leadership Group. Psychotropic Medication Toolkit for Public Children Services Agencies. Ohio Department of Job and Family Services; 2014. Accessed February 17, 2021. https://jfs.ohio.gov/ocf/2018-APSR-AppendixB2.stm

30.

National Collaborative for Innovation in Quality Measurement. National Committee for Quality Assurance. Accessed May 31, 2019. https://www​.ncqa.org​/hedis/reports-and-research​/national-collaborative-for-innovation-in-quality-measurement/

31.

Williams R. Using the margins to command to estimate and interpret adjusted predictions and marginal effects. Stata J. 2012;12(2):308-331.

32.

Humphries A. The Rutgers CERT helps establish updated parameters for psychotropic medication use in Texas foster children. AHRQ CHAIN website. Accessed June 1, 2019. https://www​.ahrq.gov​/chain/practice-tools​/behavioral-mental-health​/foster-children-psychotropic-use​.html

33.

Metabolic Monitoring for Children and Adolescents on Antipsychotics (APM). National Committee for Quality Assurance. Accessed May 31, 2019. https://www​.ncqa.org​/hedis/measures/metabolic-monitoring-for-children-and-adolescents-on-antipsychotics/

34.

Thackeray J, Crane D, Fontanella C, Sorter M, Baum R, Applegate M. A Medicaid quality improvement collaborative on psychotropic medication prescribing for children. Psychiatr Serv. 2018;69(5):501-504. [PubMed: 29493415]

35.

Ohio Minds Matter. Understanding the use of psychotropic medications in our children. Ohio Medicaid. 2018. Accessed June 1, 2019. http:​//ohiomindsmatter.org/about

36.

Akincigil A, Mackie T, Cook S, Hilt R, Crystal S. Effectiveness of mandatory peer review to reduce antipsychotic prescriptions for Medicaid-insured children. Health Serv Res. 2020;55(4):596-603. [PMC free article: PMC7375997] [PubMed: 32567089]

37.

Barclay RP, Penfold RB, Sullivan D, Boydston L, Wignall J, Hilt RJ. Decrease in statewide antipsychotic prescribing after implementation of child and adolescent psychiatry consultation services. Health Serv Res. 2017;52(2):561-578. [PMC free article: PMC5346490] [PubMed: 28297075]

38.

Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. J Am Stat Assoc. 2010;105(490):493-505.

39.

Crystal S, Jarrin O, Rosenthal M, Hermida R, Angell B. National partnership to improve dementia care in nursing homes campaign: state and facility strategies, impact, and antipsychotic reduction outcomes. Innov Aging. 2020;4(3):igaa018. doi: 10.1093/geroni/igaa018 [PMC free article: PMC7365953] [PubMed: 32699827] [CrossRef]

40.

National Quality Forum. Measuring performance. Accessed on May 30, 2019. https://www​.qualityforum​.org/Measuring_Performance​/Measuring_Performance.aspx

41.

Centers for Medicare & Medicaid Services. Children's health care quality measures. Accessed June 1, 2019. https://www​.medicaid​.gov/medicaid/quality-of-care​/performance-measurement​/adult-and-child-health-care-quality-measures​/childrens-health-care-quality-measures​/index.html

42.

Knapp P, Chait A, Pappadopulos E, Crystal S, Jensen P, T-MAY Steering Group. Treatment of maladaptive aggression in youth: CERT guidelines I. Family engagement, assessment and management. Pediatrics. 2012;129(6):e1562-e1576. doi:10.1542/peds.2010-1360 [PubMed: 22641762] [CrossRef]

43.

Scotto Rosato N, Correll C, Pappadopulos E, et al. Treatment of maladaptive aggression in youth: CERT guidelines II. treatments and ongoing management. Pediatrics. 2012;129(6):e1577-e1586. doi:10.1542/peds.2010-1361 [PubMed: 22641763] [CrossRef]

44.

Crystal S, Akincigil A, Bilder S, Walkup J. Studying prescription drug use and outcomes with Medicaid claims data: strengths, limitations, and strategies. Med Care. 2007;45(10 Suppl):S58-S65. [PMC free article: PMC2486436] [PubMed: 17909385]

45.

Substance Abuse and Mental Health Services Administration (SAMHSA). Guidance on Strategies to Support Best Practice Prescribing of Antipsychotics in Children and Adolescents. Published March 2019. Accessed February 18, 2021. https://store​.samhsa​.gov/product/guidance-on-strategies-to-promote-best-practice-in-antipsychotic-prescribing-for-children​/PEP19-ANTIPSYCHOTIC-BP

46.

Meinhofer A, Angleró-Díaz Y. Trends in foster care entry among children removed from their homes because of parental drug use, 2000 to 2017. JAMA Pediatr. 2019; 173(9):881-883. doi:10.1001/jamapediatrics.2019.1738 [PMC free article: PMC6632152] [PubMed: 31305925] [CrossRef]

47.

Winstanley EL, Stover AN. The impact of the opioid epidemic on children and adolescents. Clin Ther. 2019;41(9):1655-1662. doi:10.1016/j.clinthera.2019.06.003 [PMC free article: PMC7017799] [PubMed: 31303278] [CrossRef]

Related Publications

1. Mackie T, Olfson M, Crystal S, Cook S, Akincigil A. Antipsychotic use among youth in foster care enrolled in a specialized managed care organization intervention. J Am Acad Child Adolesc Psychiatry. 2020;59(1):166-176.e3. doi:10.1016/j.jaac.2019.04.022 [PubMed: 31071384] [CrossRef]
2. Rettew DC. Editorial: Beyond the prior authorization: multimodal efforts to reduce antipsychotic medication prescribing in foster care youth. J Am Acad Child Adolesc Psychiatry. 2020;59(1):43-44. [PubMed: 31731012]
3. Akincigil A, Mackie TI, Cook S, Hilt RJ, Crystal S. Effectiveness of mandatory peer review to reduce antipsychotic prescriptions for Medicaid-insured children. Health Serv Res. 2020;55(4):596-603. doi:10.1111/1475-6773.13297 [PMC free article: PMC7375997] [PubMed: 32567089] [CrossRef]
4. Mackie TI, Kovacs KM, Simmel C, Crystal S, Neese-Todd S, Akincigil A. A best-worst scaling experiment to identify patient-centered claims-based outcomes for evaluation of pediatric antipsychotic monitoring programs. Health Serv Res. 2021;56(3):418-431. doi:10.1111/1475-6773.13610 [PMC free article: PMC8143685] [PubMed: 33369739] [CrossRef]
5. Simmel C, Bowden, C, Neese-Todd, S, Hyde, J, Crystal, S. Antipsychotic treatment for youth in foster care: perspectives on improving youths' experiences in providing informed consent. Am J Orthopsychiatry. 2021;91(2):258-270. [PMC free article: PMC8404191] [PubMed: 33983774]

Work on this project contributed to the development of the following guidance:

6. Substance Abuse and Mental Health Services Administration (SAMHSA). Guidance on Strategies to Support Best Practice Prescribing of Antipsychotics in Children and Adolescents. Published March 2019. https://store​.samhsa​.gov/product/guidance-strategies-promote-best-practice-antipsychotic-prescribing-children-and

Acknowledgments

The authors are grateful to the project participants who shared their perspectives, experiences, and wisdom with the project team. We also appreciate the expert assistance of the project SAB, colleagues at Youth MOVE National, Johanna Bergan, Brie Masselli, and Kristin Thorp, and Irene Clements of the NFPA for their contributions to multiple aspects of this project. In addition, this work would not have been possible without commitment and support from the 4 study states and state members of the SAB. We very much appreciate the insight and experience they kindly shared with us.

Research reported in this report was funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#IHS-1409-23194). Further information available at: https://www.pcori.org/research-results/2015/examining-effects-four-state-policies-monitor-use-antipsychotics-children

Appendix

Aim 3 Analytic and Methodologic Criteria by State and Metric (PDF, 132K)

Suggested citation:

Crystal S, Mackie T, Simmel C, et al. (2021). Examining Effects of State Policies to Monitor Mental Health Medicines for Children in Foster Care. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/02.2021.IHS.140923194

Disclaimer

The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.