An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Dror Ben-Zeev, Rachel M. Brian, Geneva K. Jonathan, Lisa Razzano, Nicole Pashka, Elizabeth A. Carpenter-Song, Robert E. Drake, and Emily A. Scherer.
Author Information and AffiliationsSerious mental illness (SMI), such as schizophrenia and bipolar disorder, affects approximately 4% of the US population and profoundly affects individuals, their families, and communities. Mobile Health (mHealth) approaches that use mobile phones in support of health care can help overcome some of the barriers associated with clinic-based care. Whether mHealth interventions can serve as standalone treatments, effectively engage individuals with SMI in remote care, and produce clinical outcomes that are comparable to clinic-based interventions is unknown.
mHealth approaches that use mobile phones to deliver interventions can play an important role in improving access to care for people with serious mental illness. The goal of this study was to evaluate how an mHealth intervention (FOCUS) performs against a traditional group-based treatment (Wellness Recovery Action Planning [WRAP]). The specific aims of the study were to (1) evaluate and compare the willingness and ability of individuals with SMI to enroll in the 2 illness self-management interventions, (2) examine and compare participant engagement and satisfaction with both treatments, and (3) examine and compare patient outcomes following participation in the interventions.
We conducted an assessor-blind, 2-arm, randomized controlled intervention superiority trial between June 2015 and September 2017 in partnership with Thresholds, a large community mental health agency that provides services to people with serious mental illness living in the Midwestern United Sates. The participants were 163 predominantly minority clients with long-term, serious mental illnesses (49% with schizophrenia/schizoaffective disorder, 28% with bipolar disorder, and 23% with major depressive disorder). The outcomes were engagement throughout the intervention period, satisfaction posttreatment, and improvement in clinical outcomes (assessed at baseline, posttreatment, and 6-month follow-up). Our primary clinical outcome was general psychopathology, which is most appropriate for the range of clinical groups represented in the sample of people with serious mental illness. Secondary clinical outcomes included depression, psychosis, recovery, and quality of life. Individuals were randomized (1:1 ratio) into FOCUS (n = 82) or WRAP (n = 81). Interventions were deployed for a period of 12 weeks, using cycles of 8 cohorts of participants assigned to individual FOCUS or group-based WRAP over parallel periods. Participants were not monetarily incentivized to engage in interventions but were compensated for completing assessments. The study was registered at ClinicalTrials.gov (https://clinicaltrials.gov/ct2/show/NCT02421965).
Participants assigned to FOCUS were more likely to commence treatment (90% vs 58%) and remain fully engaged in 8 weeks of care (56% vs 40%) than those assigned to WRAP. Patient satisfaction ratings were comparably high for both interventions. Participants in both groups improved significantly and did not differ on clinical outcomes, including general psychopathology and depression. We saw statistically significant improvements in recovery within the WRAP group posttreatment, and we saw significant improvements in recovery and quality of life within the FOCUS group at 6-month follow-up. There were no statistically significant differences between groups in retention of gains over time.
Both the FOCUS and WRAP interventions produced significant gains in predominantly minority clients with severe and persistent mental illnesses. The mHealth intervention (FOCUS) showed superior treatment commencement and patient engagement and produced patient satisfaction and clinical outcomes that were not statistically different from those produced by a widely used clinic-based group intervention for illness management (WRAP). If clinicians or patients are given the option to select from these 2 treatment options, the findings of our research can directly inform their decision-making. Broadly, this study supports the notion that mHealth can play an important role in 21st century mental health care. Contemporary mobile phone “smart” functionalities enable these devices to serve as much more than staid information repositories. Audio and video media players, graphics displays, interactive capabilities, bidirectional calling and texting, and internet connectivity create new opportunities to engage patients with both automated resources and human supports. Policymakers contemplating whether mHealth should be recognized as a viable (and potentially billable/reimbursable) service will be encouraged to see our findings, which suggest that some mHealth interventions warrant consideration, as they produce outcomes that are comparable to more time-/labor-/resource-intensive clinic-based care.
We did not include a usual-treatment comparator arm. We are unable to conclude that the clinical outcomes reported in the study are a direct result of the interventions deployed, rather than the passage of time, artifacts related to involvement in research, or treatment from outside the research.
We developed measures of engagement and satisfaction for this study that have not been evaluated and validated in previous research. Because the study was powered to detect differences in the full sample between treatment groups, exploratory analyses have reduced power and thus should be interpreted with caution.
Serious mental illness (SMI), such as schizophrenia and bipolar disorder, affects approximately 4% of the population in the United States1 and profoundly affects individuals, their families, and communities. Individuals with SMI contend with distressing symptoms and are at increased risk for a range of problems, including homelessness, medical complications, arrest, incarceration, victimization, self-injury, and suicide.2 By definition, SMI results in functional impairments that significantly interfere with such life activities as work, independent living, and self-care.3 SMI is one of the leading causes of disability worldwide.4 Individuals with SMI typically experience periods of illness exacerbation and periods of partial of complete remission.5,6 With appropriate supports, people with SMI can lead rewarding and productive lives, even in the context of ongoing symptoms.7,8
Mental health clinics are offering more self-management interventions that increase and lengthen the periods in which people with SMI remain healthier and functional.9-14 These interventions can help people adhere to treatment regimens, reduce the severity and distress associated with symptoms, avoid hospitalization, increase self-esteem, and improve perceived recovery.15 However, the barriers associated with clinic-based models of care may limit the benefits of these interventions. When individuals with SMI experience symptom exacerbations (arguably, when they need illness management support the most), they may avoid going to a clinic or interacting with others, for reasons such as the clinic's distance from their residence or hours of operation,16,17 the stigma associated with seeking care,18 or dissatisfaction with the quality of services.19-21 If these individuals do decide to seek care, they must contend with these challenges.
Mobile Health (mHealth) approaches that use mobile phones in support of health care can help overcome some of the barriers associated with clinic-based care. Recent research conducted by our investigative team, and by others, has found that mobile phone ownership among people with SMI is not dramatically different from the general population: 72% to 97% of people with SMI in the United States own and use mobile phones, including smartphones.22-24 Even unsheltered homeless individuals prioritize owning mobile phones, which help them stay connected with others and access resources.25,26 Mobile phones are carried on the person, are typically turned on, and allow for bidirectional communication and on-demand access to resources.27,28 Thus, mobile phones can facilitate delivery of time-sensitive health information and point-of-care illness self-management resources wherever and whenever patients need them most. Smartphones can house mHealth applications designed to be used by patients and providers for diagnostics, behavioral prompts, reminders, and illness monitoring and self-management programs that extend far beyond the limits of a physical clinic.27,29 The combination of these factors makes mobile phones an ideal tool to support patient-centered care.
Mobile phones are ubiquitous, even among people with SMI who often have limited access to resources.22,30,31 Research across several continents has shown that most adults with SMI are interested in using their mobile phones as instruments for self-management.32 Early efforts at using mHealth for SMI have produced promising outcomes in terms of feasibility, acceptability, and preliminary efficacy.33-38 Whether mHealth interventions can serve as standalone treatments, effectively engage individuals with SMI in remote care, and produce clinical outcomes that are comparable to clinic-based interventions is unknown.
The primary objective of this study was to compare smartphone-delivered mHealth (ie, the FOCUS mobile intervention) to clinic-based self-management (ie, Wellness Recovery Action Planning, or WRAP, groups) for people with SMI. We evaluated differences in patient engagement, satisfaction, and clinical outcomes between treatment groups. FOCUS and WRAP are similar in that both interventions are recovery oriented, use an array of empowerment and self-management techniques, involve similar intervention periods, and have empirical findings suggesting they are engaging and beneficial to people with SMI.34,39-43 The differences between these approaches (ie, accessed in one's own environment vs administered in center, largely automated vs person delivered, on-demand vs scheduled) represent core distinctions between mHealth and clinic-based models of care. To our knowledge, this study is the first comparative effectiveness trial with a head-to-head comparison of mHealth and clinic-based interventions for people with SMI.
We developed the study protocol with individuals who have lived experience of SMI (N = 3); they contributed to project conceptualization and design (eg, initial intervention selection, development of recruitment strategies), data collection (eg, qualitative interviews, group discussions), and monitoring (eg, oversight and feedback from members of the project advisory board [PAB]).
Along with staff from Thresholds (N = 8), a large community-based agency that provides a range of services to people with SMI in the Midwestern United States (primary study site; see Study Setting for details), we established a PAB comprising people with SMI (N = 8) who met and advised the research team every 6 months throughout the study in a variety of ways. A consumer of Thresholds services assisted with the organization and another person with lived experience on the research team helped lead several meetings. The PAB provided peer advice and feedback on research procedures, materials for participants, and best ways to disseminate findings. Participants who first completed the study asked for additional resources to help obtain a cell phone after the completion of the study. This concern was then raised to the board. With the PAB's direction, we developed additional materials to provide these resources. They helped design the handouts, revised content and chose appropriate language, suggested displaying information in different ways to make it more understandable, and provided feedback on additional resources they knew of for obtaining a phone. The PAB also developed additional materials to display findings in the most understandable way to other patients. They also provided feedback on best places to disseminate these findings. After the materials were developed, they were reviewed and approved by the University IRB and given to participants when they completed the study.
Prior to the actual funding of the project, formative discussions began with members with lived experience and mental health experts. Biweekly strategy meetings were conducted between the research team and Thresholds to assess areas of need and interest as well as resources and capacities. Stakeholders at Thresholds provided invaluable education about current practices that have informed the selection of the comparator intervention (WRAP), expectations and projections for participant recruitment/enrollment, and knowledge about the existing member advisory board.
Research assistants (RAs; N = 2)—1 with lived experience, the other with an immediate family member with SMI—participated in qualitative data collection and analysis, lending their mental health consumer perspectives to the implementation and interpretation of the qualitative segments. The RA with direct lived experience also served as the primary technical trouble-shooting lead, not only assisting the Thresholds team with its technical needs, but also meeting with participants when outreach was necessary during a needed major update.
The objective of the study was to compare 2 illness self-management interventions for SMI: a widely used44 clinic-based protocol (WRAP) and a novel mHealth smartphone intervention (FOCUS). The specific aims of the study were to (1) evaluate and compare the willingness and ability of individuals with SMI to enroll in the 2 illness self-management interventions, (2) examine and compare participant engagement and satisfaction with both treatments, and (3) examine and compare patient outcomes following participation in the interventions.
We constructed the study as a randomized controlled trial. We collected data using a comprehensive mixed methods quantitative and qualitative approach; 163 participants with SMI currently receiving Thresholds services were randomized (1:1 ratio) to receive 12 weeks of WRAP or FOCUS. We chose a comparative effectiveness randomized controlled trial as the best design to compare 2 interventions while controlling for potential allocation bias. The mixed design provides quantitative data to support outcome differences statistically, while also going in depth through qualitative interviews and focus groups to “reach the parts other methods cannot reach” and better understand the “black box” of interventions.45,46
We conducted the study in partnership with Thresholds, the largest and oldest mental health agency in the Midwest, annually serving more than 5500 individuals with SMI in the greater Chicago, Illinois, area. Approximately 61% of Thresholds clients have a diagnosis of schizophrenia or schizoaffective disorder, and 30% have a diagnosis of a major mood disorder (eg, bipolar disorder, major depressive disorder). The agency provides access to medical care and psychiatric treatment, job readiness and work placement, educational opportunities, and housing. It offers comprehensive services, including case management, residential care, psychiatric services, supported employment, housing, and illness self-management training. Chicago is ethnically diverse, as are Thresholds' clients: Approximately 62% are male, more than half identify as Black/African American, and approximately 10% self-identify as Hispanic. The average client age is 43 years (range, 16-86 years), and the overwhelming majority of clients receive Medicaid. Thresholds has more than 18 centers in Illinois, including 3 new urban peer support centers that offered the project's clinic-based WRAP services.
Participants with SMI were initially identified using electronic health records (see appendix for a complete list of diagnostic codes used) and then recruited by 20 clinical teams at 3 Thresholds locations in Chicago. Clinical staff approached candidates to describe the project and provide informational handouts with a contact number for the research team. Interested clients called study staff to learn more and undergo a brief phone screening. Reasons for refusal were carefully tracked and reported back to the site coordinator to enter in the tracking data set. Suitable candidates were invited to attend a more comprehensive in-person evaluation meeting. After enrolling in the study, eligible participants met with a blinded clinical rater and were administered paper-based assessments. Data from the assessments were entered into the data set.
After baseline assessments were conducted, an envelope with study arm allocation and next steps was given to the participant to open on their own. Participants in the WRAP group were given their facilitators' contact information, session schedule, and a laminated card with directions to the clinic. All other materials (eg, WRAP binder, handouts) were distributed in the first WRAP session. During the first FOCUS meeting, FOCUS participants were trained by the mHealth support specialist on how to use the functions of the smartphone (eg, using a touchscreen, call, text) and different features of the FOCUS intervention. Participants then engaged in 3 FOCUS intervention sequences on a demonstration device (see Intervention section for more details), with assistance from the mHealth specialist as necessary. Once participants demonstrated their proficiency (eg, worked through smartphone screens, used media players, responded to audio prompts), they engaged in a shared decision-making interview with their mHealth support specialist to select the most relevant FOCUS intervention targets from the 5 broad domains offered, and the most suitable prompting schedule (eg, prompted to engage in a mood evaluation in the mornings if someone typically wakes up feeling depressed) and frequency (ie, up to 3 prompts daily).35 Participants then received an Android smartphone and charger with a national calling and data plan, along with the FOCUS intervention downloaded and calibrated to the selected treatment targets. The mHealth specialist would provide weekly calls with participants. All participants received a call from study RA 3 days after enrollment to ensure that the smartphone and mHealth intervention were working properly (FOCUS group) or to verify that they knew how to get to group sessions (WRAP group).
Study inclusion criteria were (1) chart diagnosis of schizophrenia, schizoaffective disorder, bipolar disorder, or major depressive disorder; (2) aged 18 years or older; and (3) a rating of 3 or lower on 1 of 3 items comprising the Domination by Symptoms factor from the Recovery Assessment Scale (RAS),47,48 indicating a need for services. Exclusion criteria were (1) hearing, vision, or motor impairment affecting operation of a smartphone (determined using a demonstration device); (2) less than fifth-grade English reading ability (determined using the reading section of the WRAT-449); and (3) having received a WRAP or FOCUS intervention in the past 3 years.
Participants were paid $30 for each assessment, but they were not compensated for engagement in the intervention. Of the group of 163 we assessed, we selected a subset of 31 participants for qualitative interviews. We recruited participants to reflect community race/ethnicity demographics and diversity of engagement (eg, high/low) in the interventions. The subset of participants who participated in qualitative interviews received $30 per interview. We recruited an additional group of 4 Thresholds practitioners to participate in focus groups at the end of the patient data collection phase. Practitioners received $30 each for participation. Due to the very small sizes of the focus groups (n = 2 per intervention arm), we are unable to draw well-represented findings, so they are not included in the final report.
To gain a deeper understanding of the experiences of people with SMI through both programs, in-depth interviews were conducted. Both high and low engagers (patients who chose to engage and to not engage in each program) were size exist for qualitative research,50 some research suggests that saturation is typically achieved between 12 and 18 interviews.51 With this in mind and because we purposively sampled to include a range of engagement, our goal was to enroll 30 participants in the qualitative study.
Our team worked collaboratively with mental health researchers and clinicians to identify the interventions to be compared in this study, converging on 2 interventions that are strongly patient centered. The mHealth intervention, FOCUS, was designed and developed in collaboration with patients and community provider stakeholders. WRAP was developed by mental health consumers and is widely disseminated as the standard of care in community settings; it is one of the most widely disseminated illness self-management programs.42,52 Both WRAP and FOCUS are typically conducted over an 8- to 10-week period. To accommodate for holidays or technical problems, we selected a 12-week treatment time frame.
FOCUS34,52 is a multimodal, smartphone-delivered intervention for people with SMI, comprising several elements. The first element is the FOCUS application (app), designed for people who may have salient psychiatric symptoms, cognitive impairments, and limited education/low literacy. The app includes preprogrammed daily self-assessment prompts as well as on-demand functions that can be accessed 24 hours a day, as long as the phone is charged. Self-management content was adapted from psychosocial intervention strategies targeting 5 broad domains: voices (ie, coping with auditory hallucinations via cognitive restructuring, distraction, guided hypothesis testing), mood (ie, managing depression and anxiety via behavioral activation, relaxation techniques, supportive content), sleep (ie, sleep hygiene, relaxation, health and wellness psychoeducation), social (ie, cognitive restructuring of persecutory ideation, anger management, activity scheduling, skills training), and medication (ie, behavioral tailoring, reminders, psychoeducation). Content can be accessed as brief video or audio clips or sequences of digital screens with written material coupled with visual displays.39
The second FOCUS element is the clinician dashboard. User responses to daily self-assessments are, who met with each FOCUS user individually at the beginning of treatment to provide him or her with a smartphone with an active data plan and FOCUS installed. The mHealth specialist worked with each participant to identify relevant content areas to set up his or her daily prompts, and guided him or her through a brief tutorial on how to use the phone (eg, how to receive and make calls, use the touchscreen, set the volume) and engage in the intervention (eg, responding to prompts, accessing on-demand features, exploring modules). In every subsequent week, the mHealth specialist called FOCUS users to check in, solve technical difficulties, and explore how FOCUS strategies could be applied to their personal recovery goals (10- to 15-minute call). The mHealth specialist reviewed the clinician dashboard to inform these weekly calls. In the final week, participants met with the mHealth specialist again to debrief and to return the smartphone.
Wellness Recovery Action Plan14 is a widely used41 group self-management intervention led by trained facilitators with lived experience of mental illness. Sessions follow a sequenced curriculum, and specific group discussion topics and examples draw from the personal experiences of the participants and cofacilitators in attendance. Although each WRAP group is different, the model emphasizes individuals equipping themselves with “personal wellness tools,” focusing on recovery concepts (eg, hope, personal responsibility, self-advocacy), language (eg, person-first recovery language), development of a WRAP (eg, establishing a daily maintenance plan, identifying and responding to triggers and early warning signs), and encouraging positive thinking (eg, changing negative thoughts to positive thoughts, building self-esteem, suicide prevention, journaling). Facilitators incorporate these tools into a written plan, which includes daily use, identification of triggers and methods to avoid them, warning signs and response options, and a crisis management plan. Compared with services as usual, WRAP produces significantly greater reduction in psychiatric symptoms, including depression and anxiety, and significantly greater improvement in recovery, hopefulness, and quality of life.41,42
WRAP facilitators and the Thresholds director of recovery participated in advanced WRAP facilitator training provided by trainers from the Copeland Center, the hub for WRAP and lead training authority in the world. Two facilitators led WRAP sessions over 12 weeks, with sessions lasting 2 hours. After each WRAP session, facilitators met with the director of recovery to review activities, plan for the next session, and examine model fidelity to ensure session content adhered to Copeland Center guidelines. Participants who missed a weekly meeting had opportunities for makeup sessions in person or over the phone that same week.
We considered participants as commencing treatment if they used FOCUS 1 time or attended 1 WRAP session. We calculated engagement in treatment for each participant weekly using their FOCUS use data (logged automatically by the software) or WRAP session attendance (logged weekly by WRAP facilitators). We considered participants in the FOCUS arm “engaged” if they used the app on at least 5 of 7 days a week (ie, approximately 70%). A FOCUS “use” event was recorded as such only if, following a prompt, participants elected to engage in a clinical status assessment or if they self-initiated one of the FOCUS on-demand tools outside of their prompting schedule. We considered participants in the WRAP arm “engaged” if they attended at least 60 minutes of the scheduled 90-minute group session (ie, approximately 70%) or completed a makeup session in the same week.
Patient willingness to enroll is an important indicator of whether patients (rather than their clinicians) find the intervention appealing. Patient engagement is important because if individuals find interventions burdensome, complex, or unhelpful, they may disengage from services prematurely, rendering even potentially potent interventions ineffective. Patient satisfaction will help inform how future interventions are developed and deployed, that is, how interventions can be improved in content or structure to increase satisfaction.
We measured satisfaction as the sum of 5 self-report items completed during the 3-month, postintervention assessment. Participants rated the following statements using a 7-point rating scale (1 = strongly disagree to 7 = strongly agree): I am satisfied with the treatment program; the treatment program helped me feel better; the treatment program was not interactive enough (reverse scored); I enjoyed the treatment program; I would recommend the treatment program to a friend.
We assessed patient outcomes with valid and reliable measures of symptoms, recovery, and quality of life. These are the outcomes that people with SMI notice and care about. Our primary clinical outcome was general psychopathology, which is most appropriate for the different clinical groups represented in the sample of people with SMI. We measured general psychopathology with the Symptom Checklist K-9 (SCL-K9)54—a brief version of the Symptom Checklist-90R56 capturing several domains of mental health (eg, anxiety, somatization, hostility, paranoid thinking, psychoticism) that provides a single global rating of severity.55,56 Secondary clinical outcomes include depression, psychosis, recovery, and quality of life. We assessed depression with the Beck Depression Inventory-Second Edition (BDI-II),57 which includes 21 items rated on a 4-point scale, summed for a total depression severity score. We assessed psychosis with the Psychotic Symptom Rating Scales (PSYRATS).58 PSYRATS includes dimensions of auditory hallucinations (eg, frequency, duration, loudness, distress) and delusions (eg, preoccupation, conviction, disruption). We assessed recovery using the RAS,47,48 a 24-item measure assessing 5 recovery factors using a 5-point Likert scale: personal confidence and hope, willingness to ask for help, goal and success orientation, reliance on others, and domination by symptoms. We assessed quality of life (QOL) as the total of 6 items focusing on one's personal evaluation of one's life, self, family, time spent with family, time spent with others, and participation in activities. Participants responded on a 7-point “delighted-to-terrible” scale. Assessors who were trained and supervised by licensed clinical psychologists with extensive experience in their administration among people with SMI administered clinical outcome assessments. Challenges in administration or scoring were brought up and resolved during weekly project team meetings.
We conducted the randomized controlled trial between June 2015 and September 2017. Assessment 1 occurred after candidates were enrolled, but before allocation to an intervention arm. Assessment 2 took place upon completion of the intervention (3 months). Assessment 3 took place 3 months after interventions were completed (6 months after enrollment), to examine whether clinical gains had been maintained (see Figure 1).
Participation Timeline.
Research staff at Thresholds used multiple strategies to engage with members participating in research programs. Enrolled participants were contacted directly using personal phone numbers. Participants were also contacted through case managers and other clinical providers regarding appointments and communication with interviewers. Efforts to engage and retain members during their participation in research programs are best reflected in study response rates.
Several strategies were used to document reasons participants discontinued participation. If members were unable to be contacted by phone, clinical service providers were notified and asked for assistance with locating members or new contact information. In the event service providers were unable to locate members, electronic health records were reviewed to determine if members were no longer receiving services from the agency or were in institutional care (eg, hospitalized, in jail), or if other factors made them unavailable for follow-up.
We designed the study as a superiority trial. We aimed to detect a medium effect (defined as f = 0.24) in difference between groups in change in clinical outcome from baseline to 3 months, with 80% power (ɑ = .05). This power is achieved with 72 participants per group. Missing data were minimal. We imputed values that were missing (due to participant refusal to respond or rater error) with the average scores for responses given from all participants. Descriptive statistics characterized the sample with respect to demographic and health information at baseline, and we used t tests and χ2 tests (for continuous and categorical variables, respectively) to compare the groups on these variables. We evaluated engagement using χ2 tests, and treatment satisfaction via t tests. For treatment comparisons among clinical outcomes, we used mixed-effects models, including treatment condition, study time point (ie, baseline, 3-month/end of treatment, 6-month follow-up), and an interaction term for treatment condition by time. We fit linear mixed models59 for all outcomes except PSYRATS, which we modeled via a nonlinear Poisson hurdle mixed model, which estimates a logistic model for probability of a count >0 (likelihood of experiencing symptoms) as well as a Poisson model for mean symptom ratings if any symptoms were experienced.60 We modeled PSYRATS in this way because of the skewed nature and zero-inflation (eg, 64% of individuals had a score of 0 at baseline) observed in this outcome, which made linear models inappropriate.
For primary and secondary clinical outcomes, we were interested in the difference among intervention groups for changes between baseline to end of treatment (ie, the treatment effect). Treatment effect is tested by the significance of the treatment × time point interaction term for the 3-month/end of treatment time point, using baseline as the reference time point. A secondary comparison addressed changes from 3-month/end of treatment to the 6-month follow-up, using contrast statements comparing the treatment × time point interaction term for 3-month vs 6-month follow-up. We included a random individual-level intercept in the model to account for repeated observations within an individual. We used an intent-to-treat analysis including all randomized individuals. We evaluated whether the treatment effect differed by diagnosis group by including a diagnosis main effect, interactions with time point and treatment condition, and a 3-way interaction (diagnosis × time × treatment). Among individuals receiving FOCUS, we examined whether baseline demographic, treatment use, or health information was associated with clinical outcomes. We performed this subgroup analysis via mixed-effects models. We performed all analyses using SAS version 9.4.
Before interviewing participants, interview guides were developed for engaged and nonengaged participants in both arms (see the appendix). All interviews were audio-recorded, then transcribed by a transcription company. Three members of the research team who had conducted the interviews reviewed the first round of interviews and independently generated a list of initial concepts and categories. Involving multiple analysts is an accepted check and balance on rigor in qualitative research because it allows for multiple perspectives. Based on this review, the study aims, and domains of the interview guide, a codebook of 37 initial codes was developed and used to code the transcripts using a thematic analytic approach.61 Provisional themes identified in the early interviews were used to identify areas of investigation in subsequent interviews. Through continued immersion in the data set, 8 additional codes were established after the research team had finalized the codebook. Coding was led by 1 researcher, with ongoing supervision and review by a qualitative researcher as a check on credibility and trustworthiness. We used Dedoose, a qualitative analytic software program, to manage and code the data. We generated code excerpt reports for the following domains to provide insight into patient engagement and satisfaction with the interventions as well as perceived impact of the interventions: (1) indicators of acceptability of the interventions, (2) indicators of unacceptability of the intervention, (3) challenges with the interventions, and (4) impact of the interventions. Two researchers reviewed and annotated the code excerpt reports to produce analytic memos that identified patterns across codes. The 2 researchers shared and reviewed analytic memos as a check on credibility of interpretations. Analytic memos were subsequently revised and refined into data summary reports that outline main findings and include emblematic quotations. Saturation of themes had been met at 20 interviews.
Initially, we identified N = 255 individuals as possible candidates for the study (Figure 2). A quarter of this group did not meet inclusion criteria: 19 failed an initial phone screening, 18 did not accept the terms of the study (eg, randomization to care vs based on personal preference, wanting more compensation), 13 failed the in-person screening, 13 did not have the correct diagnosis, and 2 were no longer receiving services at the agency. Of the N = 163 individuals consented, 100% were randomized to 1 of the 2 study intervention arms and completed baseline interviews. Of these, 92% (n = 149/162) completed study interviews at the 3-month follow-up; 89% (n = 144/161) completed 6-month follow-up interviews. Within intervention arms, approximately 91% (n = 75/82) of participants randomly assigned to FOCUS completed 3-month follow-up interviews; 91% (n = 74/81) of WRAP participants completed interviews at 3 months. At 6-month follow-up, 90% (n = 74/82) of FOCUS participants completed interviews, with 86% (n = 70/81) of WRAP participants completing interviews. Reasons why people were lost to follow-up included lost to agency, withdrew from study, medically unavailable, incarcerated, and no longer received services at study site (closed to agency).
Study CONSORT Diagram.
Participants were on average 49 years old, 59% were male, and 65% were African American. The sample included 49% individuals with schizophrenia or schizoaffective disorder, 28% with bipolar disorder, and 23% with major depressive disorder. Participants differed between treatment groups only in that significantly more participants randomized to FOCUS (73% vs 57%; χ2[1] = 4.81; P = .03) had previously used a smartphone. Table 1 summarizes participant characteristics.
Participant Demographic and Health Characteristics at Baseline.
The interview sample was split across participants assigned to FOCUS (52% [n = 16]) and participants assigned to WRAP (48% [n = 15]). Most individuals who were selected were higher engagers (68% [n = 21]) vs low engagers (32% [n = 10]). The mean age of the sample was 49.5 (SD, 11.6) years. Most were male (65% [n = 20]), were African American or Black (61% [n = 19]), and had a diagnosis of schizoaffective disorder (45% [n = 14]). Most reported unemployment (94% [n = 29]), had a high school diploma or higher (74% [n = 23]), and lived independently (58% [n = 18]). Most had no hospitalizations in the past year (74% [n = 18]); however, most had at least 6 hospitalizations over their lifetime (65% [n = 20]). Although most had some prior smartphone experience (68% [n = 21]), only about half actually owned a smartphone (52% [n = 16]).
Following randomization, 90% (95% CI, 82%-96%) of participants assigned to FOCUS commenced use of the mHealth app, but only 58% (95% CI, 47%-69%) of people assigned to WRAP attended a group session (χ2[1] = 22.11; P < .001). FOCUS group participants were also more likely than the WRAP group to engage in treatment for at least 8 weeks (56% [95% CI, 45%-67%] vs 40% [95% CI, 29%-51%]) (χ2[1] = 4.50; P = .03). The groups did not differ in percentage of people fully engaging in all weeks of treatment (26% [95% CI, 17%-36%] vs 22% [95% CI, 14%-33%]). Averaging across all participants assigned to FOCUS, in the first week of the intervention participants used the app on 5.4 (SD, 2.4) days, in the third week on 4.6 (SD, 2.7) days, in the sixth week on 4.3 (SD, 2.7) days, in the ninth week on 3.9 (SD, 2.7) days, and in the last week on 3.8 (SD, 2.9) days. In the first week of WRAP, 48% of assigned participants attended the weekly meeting, in the third and sixth weeks 42% attended, in the ninth week 36% attended, and in the last week 28% attended. Figure 3 shows participant engagement throughout the intervention period.
Number and Percentage of Participants Fully Engaged in Each Treatment at Different Stages of the Interventions.
In total, 148 participants completed satisfaction ratings at the end of their treatment. Mean posttreatment satisfaction ratings were similar between arms: 25.7 (95% CI, 24.8-26.5) out of a possible 35 for FOCUS and 25.5 (95% CI, 24.6-26.4) for WRAP (t[1] = –.31; P = .76); mean difference was 0.18 (95% CI, −1.02 to 1.39). Figure 4 shows average responses for individual satisfaction items. No adverse events were reported for participants in either intervention arm.
Average Participant Satisfaction Ratings Following Treatment.
The groups did not differ in change from baseline to 3 months/postintervention on primary and secondary clinical outcomes (ie, no “group” effect). Mean difference between groups in change from baseline was for SCL-9, −0.58 (95% CI, −2.68 to 1.51); for BDI-II, −0.43 (95% CI, −3.47 to 2.61); for RAS, −0.63 (95% CI, −3.68 to 2.43); and for QOL, −0.43 (95% CI, −2.15 to 1.30). Primary analyses found no significant differences in clinical outcomes between diagnostic groups. Exploratory analyses found within-treatment group changes. Improvement in the primary clinical outcome SCL-9 was seen for FOCUS (−2.73 [0.75], t[289] = −3.64; P < .001; 95% CI, −4.21 to −1.25) and WRAP (−2.14 [0.76], t[289] = −2.84; P = .005; 95% CI, −3.63 to −0.66). Improvement in BDI-II was seen for FOCUS (−2.76 [1.09], t[289] = −2.54; P = .01; 95% CI, −4.90 to −0.62) and WRAP (−2.33 [1.10], t[289] = −2.13; P = .03; 95% CI, −4.49 to −0.18). Improvements in RAS were seen for WRAP (2.44 [1.10], t[288] = 2.21; P = .03; 95% CI, 0.27– 4.62), but not for FOCUS. There were no significant within-group differences in PSYRATS or QOL scores.
The treatment groups did not differ on changes from 3 months/postintervention to the 6-month follow-up. Mean difference between groups in change from 3 months/postintervention to 6-month follow-up was for SCL-9, 0.0042 (95% CI, −2.16 to 2.17); for BDI-II, −0.049 (95% CI, −3.18 to 3.08); for RAS 2.33 (95% CI, −0.81 to 5.47); and for QOL, 1.14 (95% CI, −0.64 to 2.91). Within treatment group, improvement in RAS was seen for FOCUS (2.74 [1.11], t[288] = 2.46; P = .01; 95% CI, 0.55-4.94). Table 2 summarizes the mean scores for primary and secondary clinical outcomes.
Clinical Outcomes Over Time in All Enrolled Participants.
Participants who engaged in FOCUS 5 days a week for 9 or more weeks found their experience using the mobile intervention to be overwhelmingly positive. Participants noted appreciating the ability to use the program for support anytime and in any place, often when their providers were unavailable. A few participants even went so far as to say that FOCUS was comparable to their clinician, except that it was in their pocket and always available:
It's like a counselor being with you 24/7. I say 24/7 because, you know, it'd be all day, most all night. – Participant ID 67
When compared with traditional mental health services, participants noted FOCUS allowed for greater privacy and instantaneous response. In addition to availability, participants thought that the app's direct and personalized feedback was better than other services they had received. Many participants shared that FOCUS provided new information about symptoms and coping mechanisms that they had not yet covered elsewhere in treatment. However, others noted having previous exposure to some strategies. Several participants felt the app could better fit personal situations and emphasized a need for more tailored and individualized care. Given that most participants used the application at least 5 days a week for the duration of the intervention, many started to long for more variety in the content they were reviewing. One participant specifically mentioned that he refrained from using the app due to the repetition.
When asked to reflect on the app's general impact, participants expressed optimism about the future. For some, the optimism was a result of learning new strategies to cope with their symptoms:
[FOCUS] upgraded me, mentally, physically, psychologically, and spiritually. It uplifted me to another level, a better understanding of my diagnosis. – Participant ID 67
The short-term goals that FOCUS established motivated some participants to pursue their aspirations—ranging from volunteer work to general self-care activities (eg, reading, housekeeping, organization, exercising). Others felt that FOCUS was an inspirational way to start their day:
FOCUS [was] a way you [could] wake up and just start your day off with something positive like, “How well did you sleep?” It was something to listen to, to make you feel motivated to get up and do whatever you need to do for that day. For me, I can say the mood [module] was definitely a plus in the morning. It would help me realize that I needed to do this or that to make myself feel better. – Participant ID 17
Participants most frequently cited their use of the app's social module (ie, cognitive restructuring of persecutory ideation, anger management, activity scheduling, skills training) as the most beneficial aspect of the program. The social module helped participants feel more open-minded toward their friends, family, and peers as well as more comfortable with taking a step back during difficult social situations to collect their thoughts. Several participants expressed that, even though their use of the intervention had discontinued, many of the lessons and strategies they learned were still relevant:
It reminded me that people are people, that they're not inherently out to get me. Sometimes stuff happens and you've got to understand they're individuals and you can't predict their behavior, you can't control [it] either. – Participant ID 10
The FOCUS prompts (automated messages sent by the app to engage) helped serve as helpful reminders, but some participants found the reminders a little annoying. Several other participants shared that although they no longer had access to the app, these reminders had helped them build a routine to follow in their day-to-day lives.
Some participants—especially those with no prior experience—found using the device difficult. However, most reported FOCUS was easy to use. Participants also enjoyed access to other applications on the phone such as the internet, FM radio, and local news. Others had minor technical issues within the application itself and did not have the technology skills to resolve the problem on their own. However, participants mentioned support from the mHealth specialist (ie, human support) as another advantage and one unlike other technical adoption support experiences they have had in the past.
Those who did not engage in FOCUS for 9 or more weeks reported that they did not need additional psychiatric support or they were unsure of how to use the technology. Overall, participants occasionally missed checking into the app when they were busy engaging in other activities (eg, school, church, community service) or because of technical issues with the application; however, they would often make up missed check-ins after they finished completing another activity. When asked to compare FOCUS to their current mental health services and identify any potential disadvantages, participants primarily felt the biggest disadvantage was a lack of face-to-face connection.
For participants who attended most or all WRAP sessions, the experience was positive. Those who attended regularly would occasionally miss sessions due to weather or conflicting appointments. Overall, participants found in WRAP a supportive community of individuals with shared experiences of mental illness. Participants described feeling “welcomed as part of a group.” Many noted that WRAP made them realize that they “weren't alone.” Shared experiences were the basis for opening up to the group. In some cases, WRAP members continued to meet with one another after the program had officially ended.
Participants emphasized that WRAP was a learning experience for them, including gaining a “new perspective” on mental illness by interacting with peers, learning new coping strategies, learning about triggers, learning how to plan in the event of a relapse, and learning how to take steps toward recovery. One participant (ID 6) spoke poignantly of WRAP as a turning point in realizing the importance of human connection and that there is hope for recovery. Participants also appreciated the practical side of learning new skills through WRAP—for example, having a daily maintenance plan that could serve as a checklist for the day.
Most participants who engaged in WRAP sessions expressed that WRAP was different from other mental health services, including other treatment groups, in which they had participated because it was oriented toward recovery, was peer led, and was more “substantial” and “in depth” than other services. Several participants noted that the shared experience of mental illness among group members facilitated their ability to open up and share experiences that they might otherwise conceal from mental health clinicians:
Oh, like common experiences that I have with the members of the group, having mental illness, medications we've taken that we don't want to take again…. I don't think the case manager would see it as the same perspective—the case manager not having a mental illness. – Participant ID 52
This participant elaborated the point, noting the need to be careful about what is said to clinicians:
Sometimes … you can't say things around [the counselors], or … tell them what you really feel…. They'll say, “You sick?” They'll put you in the hospital…. But in WRAP group, they didn't do … that. They let you open up and discuss … in a positive way and other people listen and then they gave feedback and … we felt more relaxed about talking amongst … each other about different subjects. – Participant ID 52
Some participants also noted that, in comparison to other mental health services, WRAP encouraged them to be active participants in their own recovery:
Well, some of the other groups, it's more like a teaching way of doing it. But with the WRAP, it's more you're enacting—You're more a participant … and you're more involved with your own recovery. – Participant ID 136
Others noted a change in their ways of interacting in the world:
[Participant]: It inspired me to [want to] find other things to do other than just sitting around the house and being a monotone person, you know. I actually [want to] meet [people].
[Interviewer]: Along those same lines, what do you see as some of the changes in your life now, as a result of participating in WRAP?
[Participant]: Well, I'll say I'm more active. I'm more independent. – Participant ID 92
Those who did not attend WRAP sessions reported making that decision based on other priorities (eg, seeking housing) and treatment preferences (eg, not liking groups). Some participants who attended WRAP sessions provided insight into difficulties that arose. Most of these negative experiences involved aspects of navigating complex group dynamics. One participant felt “uncomfortable” with aspects of participants' experiences that were being shared:
Well, to be honest, I was uncomfortable somewhat … sometimes during a few of the weeks [when] different members of those other groups were going through some personal problems that they were sharing with the group. It was a little uncomfortable…. I think some of the … members were in some kind of trouble or something. – Participant ID 24
Others had difficulty managing certain personalities within the group:
There … actually was one person, in particular, who was very rude. – Participant ID 92
Another felt “called out” and “targeted” by other members of the group (participant ID 94).
A few participants expressed frustration by the lack of commitment on the part of others in the group:
Well, aside from myself, there [were] only 2 others who [were] there for the whole time … some of the people were coming every once in a while. And I just felt that if we were talking about something that was so private … it seemed like people should've been more involved…. So it seems like … some people were dropping in and out and … [with] that person … it made you question whether or not your business was being put out in the world…. Like, how … true is this person to this program? – Participant ID 92
A few individuals voiced their preference for the mHealth phone intervention—and wished they had been randomized to that study condition:
I … really don't like group sessions…. I was hoping I was getting the phone. It would have been … easier. – Participant ID 12
One participant noted that the WRAP materials—a large workbook—were somewhat impractical for applying to everyday life compared with other mental health services that seemed more immediate:
[The WRAP workbook] is too big. I can't take it with me. I can't often reflect on [it]. You don't need your … crisis plan if you're not in a crisis but … if you don't have it in front of you, how are you able to add it later? – Participant ID 6
The FOCUS mHealth intervention produced clinical outcomes and patient satisfaction ratings that were comparable to WRAP, an evidence-based self-management intervention. No adverse events were reported for participants in either intervention arm. Actively matching patient preferences to intervention type, rather than random assignment, may improve outcomes even more. FOCUS had significantly higher treatment commencement rates following allocation (90%) than WRAP (58%). This finding suggests that FOCUS was easier to initiate or more accessible. Significantly more participants fully completed 8 or more weeks of treatment in FOCUS (56%) than in WRAP (40%). However, groups did not differ in the number of participants who fully completed 12 weeks of treatment. Taken together, participants were exposed to treatment content more often and over longer intervention periods (ie, dose) via FOCUS than via clinic-based WRAP.
Satisfaction with treatment did not differ across groups. Participants provided high satisfaction ratings for FOCUS and WRAP and reported both approaches were enjoyable, were interactive, and helped them feel better. Participants in both groups strongly agreed that they were likely to recommend their interventions to others. We draw several conclusions: First, people with SMI can be satisfied with mHealth treatments that are largely automated and involve weekly remote check-in calls but minimal in-person contact. Second, while barriers related to clinic-based care might have affected treatment commencement and engagement in WRAP, they did not negatively affect participants' overall impressions of the intervention. Given that a large percentage of people in the WRAP arm did not attend any sessions, satisfaction ratings may represent a nonspecific response. Participants in both arms were not exposed to the other treatment, so their satisfaction ratings were not grounded in familiarity with an alternative. A crossover design in which participants experienced both interventions would enable more direct comparison of satisfaction.
Participants in FOCUS and in WRAP did not differ statistically in their changes in primary (general psychopathology) and secondary (depression, psychosis, recovery, quality of life) clinical outcomes. There were no statistical differences between groups in retention of gains 3 months after the conclusion of the intervention period (6-month follow-up). Exploratory analyses within treatment arms showed significant small to medium reductions in psychopathology and depression in both treatment groups. Significant improvements in recovery were seen within the WRAP group at the end of treatment and within the FOCUS group at 6-month follow-up. Symptom burden among participants, while improved in both groups, was very similar between FOCUS and WRAP over the course of the study.
This study has notable strengths. To our knowledge, it is the first randomized controlled trial examining the effects of smartphone intervention involving people with schizophrenia spectrum disorders. The comparator intervention (WRAP) is an active evidence-based treatment. Both interventions were introduced at the study site at the same time, ensuring equal levels of enthusiasm and excitement among study staff and clinical personnel. Additional methodological strengths include using psychometrically sound symptom outcomes measures and random assignment, maintaining assessor blindness throughout the study, and deploying interventions as parallel cohorts to control for historical effects. Both interventions were delivered with guidance from treatment experts; the mHealth support specialist was trained and supervised via phone by the lead FOCUS developer. WRAP facilitators were trained by the Copeland Center for Wellness and Recovery (the premiere WRAP training and education center) and supervised on site by a certified advanced-level WRAP facilitator.
The portability of smartphones enables patients to take them everywhere. FOCUS users in the current study could therefore read, hear, or view self-management skills, suggestions, and demonstrations that were relevant to the challenges they encountered as they went about their daily life. Instead of having to retain and recall a clinician's suggestions, they accessed their “pocket therapist” on demand—an experience akin to a friend checking in with you.39 FOCUS users' daily self-assessments were relayed to their mHealth support specialist, who reviewed the data to better understand what users were experiencing. The specialist brought up this information in the user's weekly check-in calls, which likely strengthened the feeling that a caring individual was paying attention to their status.53 The combination of automated functions and live remote human support facilitated a therapeutic model unlike any that FOCUS participants had encountered. For some, this proved to be engaging and helpful.
The characteristics of our real-world design (eg, no artificial incentives to increase treatment engagement, using locally available staff and training programs, conducting interventions in real-world settings) and comprehensive multidimensional data collection strategy (ie, mixed methods, quantitative and assessments, purposive sampling of both high and low treatment engagers) strengthen the generalizability of our findings. However, several specific study aspects would likely not be present in general community settings: FOCUS participants were provided with a smartphone device and data plan; WRAP participants were given the opportunity to engage in individual makeup sessions flexibly, if they missed the week's group meeting; all participants were incentivized financially to participate in preintervention, postintervention, and follow-up assessment, which may have inadvertently strengthened their overall enthusiasm for the interventions or willingness to engage over time.
The findings of our research suggest the FOCUS mHealth intervention can play a useful role in the treatment of people with SMI in community settings. However, the best way to integrate FOCUS into the workflow and protocols of community practices is not clear.
Literacy and language capabilities needed to be taken into consideration for both interventions, in order to ensure accessibility for all participants. Those with severe cognitive or physical impairment may also find difficulty using both programs, since dexterity is required to complete WRAP materials and use the FOCUS application; these participants were screened out. With approximately 80 participants in each treatment arm, the study is underpowered to detect any differences in response rates to the compared interventions by participants with a particular characteristic.
The study has several limitations. We did not include a usual-treatment comparator arm. We are unable to conclude that the clinical outcomes reported in the study are a direct result of the interventions deployed, rather than the passage of time, artifacts related to involvement in research, or treatment from outside research. We developed measures of engagement and satisfaction for this study that have not been evaluated and validated in previous research. Because the study was powered to detect differences in the full sample between treatment groups, exploratory analyses have reduced power and thus should be interpreted with caution.
The findings of our research suggest the FOCUS mHealth intervention can play a useful role in the treatment of people with SMI in community settings. However, the best way to integrate FOCUS into the workflow and protocols of community practices is not clear. Systematic implementation research evaluating 1 or more well-defined mHealth implementation approaches will help fill this gap.
Delivering interventions to patients in their own environments rather than in clinics could potentially lead to substantial savings in health care–related costs. Studies focusing on health economics could help determine the true cost (to agency, provider, patient) as well as cost effectiveness of the FOCUS mHealth intervention. Given that there are no billing codes or clearly articulated reimbursement models related to mHealth, findings from this research could help inform policy makers and legislators who are contemplating how best to support and incentivize the use of novel digital health approaches in their regions. Agencies could potentially charge for the time spent reviewing the data within the FOCUS dashboard as part of remote monitoring for chronic conditions, which are Current Procedural Terminology (CPT) codes for Centers for Medicare & Medicaid Services (CMS). Demonstrating the “value” of such approaches and overall savings generated by their use would help inform such decisions.
Evidence from mHealth research will accumulate over the coming years, including additional promising results from randomized controlled trials. As more mHealth interventions prove to be engaging and clinically useful to patients with SMI, enthusiasm for their use in clinical practice will grow.22 Downstream, it will be important to ensure these mHealth technologies are deployed ethically and responsibly.62 Field trials evaluating the parameters, vulnerabilities, and constraints to patient privacy and data security associated will digital health help mitigate unintended negative outcomes in these domains.
The primary aim of this study was to empirically compare FOCUS, an mHealth intervention, and WRAP, an evidence-based self-management intervention. FOCUS produced clinical outcomes and patient satisfaction ratings that were not significantly different from WRAP and it had significantly higher treatment commencement rates following allocation (90%) when compared with WRAP (58%). These findings suggest FOCUS was easier to initiate or more accessible. Significantly more participants fully completed 8 or more weeks of treatment in FOCUS (56%) than in WRAP (40%). However, groups did not differ in the number of participants who fully completed 12 weeks of treatment.
This study supports the notion that mHealth can play an important role in 21st century mental health care.28,63 Contemporary mobile phone “smart” functionalities enable these devices to serve as much more than staid information repositories.64 Audio and video media players, graphics displays, interactive capabilities, bidirectional calling and texting, and internet connectivity create new opportunities to engage patients with both automated resources and human supports. The portability of smartphones enables patients to take them everywhere. Our findings directly address the question of whether a specific mHealth intervention (FOCUS) can produce engagement, satisfaction, and patient outcomes that are comparable to WRAP, a clinic-based intervention. If clinicians or patients are given the option to select from these 2 treatment options, the findings of our research can directly inform their decision-making. More broadly, policy makers who are contemplating whether mHealth should be recognized as a viable (and potentially billable/reimbursable) service will be encouraged to see our findings which suggest some mHealth interventions certainly warrant consideration, as they produce outcomes that are comparable to more time-/labor-/resource-intensive clinic-based care.
Prior to this study, it was unclear whether mHealth could serve as a viable alternative to existing models of care. Our study was the first comparative effectiveness trial with a head-to-head comparison of mHealth and clinic-based interventions for people with SMI. The trial demonstrated conclusively that the FOCUS smartphone intervention produces patient engagement, satisfaction, and clinical outcomes that are similar to a widely used clinic-based person-delivered intervention. For people with SMI seeking illness management support, FOCUS may be a better option than WRAP because they are more likely to start treatment and stay in treatment, without sacrificing satisfaction with care and improvement in key clinical outcomes.
Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#CER-1403-11403IC) Further information available at: https://www.pcori.org/research-results/2014/comparing-smartphone-program-peer-led-program-help-people-serious-mental
Diagnostic codes used for participant identification (PDF, 121K)
Qualitative Interview Guide for Mental Health Consumers: WRAP non-engager Guide (PDF, 86K)
Qualitative Interview Guide for Mental Health Consumers: WRAP Guide (PDF, 145K)
Qualitative Interview Guide for Mental Health Consumers: FOCUS non-engager Guide (PDF, 87K)
Qualitative Interview Guide for Mental Health Consumers: FOCUS Guide (PDF, 204K)
Ben-Zeev D, Brian RM, Jonathan GK, et al. (2019). Comparing a Smartphone Program with a Peer-Led Program to Help People with Serious Mental Illness Manage Their Symptoms. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/4.2019.CER.140311403IC
The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.
This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/
Your browsing activity is empty.
Activity recording is turned off.
See more...
