Aim

The aim of this review was to systematically acquire and synthesise evidence regarding critical components and key organisational principles of CRT services. In the review, quantitative evidence on associations between specific CRT characteristics and outcomes was prioritised. However, initial searches confirmed the belief that clear empirical evidence was likely to be limited. Literature on the views of stakeholders, including service users, carers, clinicians and service leaders, about best practice in CRTs was reviewed. In addition, the recommendations that government and expert organisations make about CRT service delivery and organisation were examined.

Methods for data collection

A systematic review was conducted. The databases MEDLINE, EMBASE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Science were searched to November 2013, without time limit or geographical restrictions. Studies were included from anywhere in the world and in any language. A further web-based search was conducted for published government and expert guidelines on CRT services; this was restricted to England. Full search terms are provided in the published report.⁵⁸

The following types of study were included:

• quantitative studies comparing outcomes between CRTs with different characteristics, or between CRTs and other service models
• national or regional surveys reporting associations between CRT characteristics and outcomes
• qualitative studies exploring the views of stakeholders, including service users, carers and clinicians, of what makes a good CRT service
• published guidelines from statutory and non-statutory bodies making recommendations for effective, good-quality CRT care in the NHS in England.

Analysis

A narrative synthesis was conducted, based on guidance from the Centre for Reviews and Dissemination.⁵⁹ Each of the above types of study was separately investigated. Statistical meta-analysis was not feasible because of the heterogeneity of included studies with regard to study aims and design, outcomes assessed, service settings and characteristics. The mixed-methods appraisal tool (MMAT),⁶⁰ a tool that allows quality rating of a variety of types of study, was used to assess the quality of individual studies.

Key findings

Sixty-nine studies and documents were included in the review. The composition and activities of CRTs varied substantially across studies. Studies also varied in quality, with the total mean score on the MMAT scale⁶⁰ indicating moderately high quality.

Overall, quantitative comparison studies and CRT surveys provide relatively little evidence regarding the specific characteristics of CRTs that are associated with positive outcomes. Many studies of CRT outcomes give relatively little detail regarding the characteristics of the CRTs, making it very difficult to draw conclusions about associations between service elements and outcomes. One comparison of two CRT models found the presence of a psychiatrist within the team to be associated with reduced admissions, and a national survey suggested an association between 24-hour opening and lower admissions. Little other relevant evidence was obtained.

Qualitative studies and government and expert guidelines provide more detailed specification of best practice in CRTs. Stakeholders emphasised accessibility, integration with other local mental health services, provision of time to talk and practical help, treatment at home when possible and continuity of care. CRT guidelines prioritised the provision of a 24-hour, 7 days per week multidisciplinary service, including a psychiatrist and medical prescriber, relapse prevention planning on discharge from the CRT and a gatekeeping role for CRTs in controlling inpatient admissions. The need for adequate staffing to meet service demands and high quality of staff training were also frequently emphasised.

Strengths and limitations

In order to establish the critical ingredients of CRTs, this review used a systematic search strategy to identify all available types of evidence. The multidatabase search for relevant research studies was supplemented with hand-searching of reference lists and contacting authors about conference abstracts. Although this search included international literature regarding CRT services and closely related models, the search for government and expert guidelines was confined to England because of resource limitations and challenges in identifying and accessing such sources.

Another limitation was the fact that because of the wide variation among studies and incomplete reporting of CRT characteristics, it was not feasible to conduct quantitative synthesis of results or compare the clinical effectiveness of CRTs across studies. Furthermore, the MMAT⁶⁰ is a relatively unsophisticated means of assessing quality, even though it is probably the best available single measure for synthesising the quality of studies using a mixture of methods. The tool treats different methodologies as equivalent and merges different components of quality into a single score, which is potentially reductive and misleading.⁶¹

The review did not exclude any papers on the basis of quality, as it was wished to fully assess the current evidence base regarding CRT services. As a result, some studies with lower quality scores were included, which may potentially compromise the strength of the conclusions. However, steps were taken to take account of the variability in the quality of included studies in our synthesis of the evidence.

Recommendations for future research

This extent to which empirically based critical ingredients of CRT good practice can be identified is constrained by the fact that only a very small number of studies address this question. Interpretation and synthesis of the studies that have been conducted are impeded by wide variations in models and outcomes investigated and lack of detailed information on service characteristics. Future studies of CRTs should provide a more comprehensive description of the CRT and comparison services, as recommended by the Consolidated Standards of Reporting Trials (CONSORT) guidelines.⁶² Although there was relatively little quantitative evidence regarding the critical ingredients associated with good CRT outcomes, there was a high degree of consensus from stakeholders and policy guidance about an optimal CRT model, broadly corroborating initial government guidance accompanying the national mandate to introduce CRTs.³ This provides a basis for development of a more highly specified CRT model and of a means to assess adherence to this model: a priority for CRT research. It would now be valuable to conduct further research into the characteristics of CRTs associated with good outcomes and good patient experiences.

Aims

A national survey of CRT managers was conducted in 2011/12. The aims of the survey were to map the provision of all CRTs in England and describe CRT organisation and service delivery, explore the teams’ adherence to key policy recommendations³ and compare CRT managers’ views on actual and desirable CRT characteristics.

Methods for data collection

Between November 2011 and July 2012, the manager of every crisis resolution/home treatment team (or equivalent) in England was invited to complete an electronic survey. If a team manager could not be reached or wished to delegate the survey, another senior member of the team was invited to participate. The survey consisted of a 90-item questionnaire, which was based on previous national surveys of crisis services^24,64 and was further refined after a pilot in four CRTs. The questionnaire covered a range of aspects of CRT organisation and service delivery, including exploring which interventions are available from CRTs. In addition, reports were elicited on local initiatives and priorities for service improvement, and views regarding ideal characteristics of CRTs. The survey could be completed online or as a telephone interview with a researcher. Data from the questionnaire were entered directly into a secure online system and were downloaded into a Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA) spreadsheet and transferred to IBM SPSS Statistics (IBM Corporation, Armonk, NY, USA) for analysis.

Analysis

There were three stages of data analysis:

1. Data regarding CRTs’ organisation and service delivery were summarised using descriptive statistics. Any free-text responses were coded to allow quantitative description of the most frequent responses.
2. When data directly related to recommendations from the original CRT guidance,³ questionnaire variables were recoded or combined in order to examine the degree to which CRTs were adhering to guidance in three domains – referral criteria and access, interventions and staffing. The extent to which CRTs adhered to guidance in each domain was reported.
3. In order to investigate any discrepancies between actual and perceived desirable CRT practice, the proportion of CRTs in which a service characteristic was present was compared with the proportion in which a service characteristic was rated by managers as fairly or very useful.

Key findings

Through service mapping, we identified a total of 218 CRTs, with some CRT provision in all 65 mental health NHS trusts in England. One hundred and ninety-two CRTs (88%) took part in the survey and 184 respondents (84%) completed at least two-thirds of the questionnaire.

Table 1 shows the composition of CRTs: nurses and psychiatrists were employed in almost all teams, but representation of other professions varied greatly from team to team.

TABLE 1

Composition of CRTs

The survey also enquired about delivered interventions (Table 2). These were most often focused on prescription and delivery of medication, rather than on psychological, social or practical interventions.

TABLE 2

Crisis response teams’ provision of different interventions

Only one-third of CRTs (62/187, 33%) reported always acting as gatekeepers to inpatient beds, as stipulated in the original policy implementation guidance.³ An even smaller proportion of CRTs (35/187, 19%) reported always attending formal assessments for compulsary admission under the Mental Health Act.²⁹ Most CRTs (186/188, 99%) were also engaged in early discharge work, identifying inpatients who could go home earlier with support from the CRT.

Team managers were asked to rate whether or not a number of aspects of service delivery were important, and to report whether or not they were actually able to provide these at present. Table 3 shows the results and indicates many discrepancies between actual and desired provision, for example in being able to provide practical support of various types to service users and in being able to spend extended periods of time with them. Thus, from the perspective of our work, the important overarching theme is that, although there are many gaps in areas such as psychological and social interventions and capacity to spend extended time with service users, such gaps do not result from these aspects of provision being seen as low priority. Rather, it appears that managers who were surveyed would have liked to deliver a much richer service than was possible in practice.

TABLE 3

Discrepancies between actual and desirable service provision

Strengths and limitations

This survey had a high response rate (88%), which gives confidence that the sample of respondents is representative of senior CRT staff in England. However, there are two main limitations. First, the use of self-reporting may have resulted in an over-reporting of CRT implementation because of social desirability bias. Second, the survey was cross-sectional in nature, thus shedding no light on changes over time.

Recommendations for future research

The survey suggested that the original policy mandate and guidance were not sufficient to achieve nationwide CRT implementation as intended. This supports conclusions from the US EBP programme³⁶ that high fidelity to good practice is unlikely to be achieved without regular structured assessment of implementation. A need was thus identified for future research to develop and test tools to support implementation and monitoring of CRTs. CRT stakeholders’ views and priorities, including those of service users and carers, should also be investigated and taken into account in developing an empirically based optimum model of CRT care. Tracking service delivery nationwide by repeated surveys over time will also be important to achieving and sustaining implementation of a CRT model matching perceived needs.

Aim

Previous research has shown that there is insufficient empirical evidence regarding the critical ingredients of CRTs (see Module 1.1) and that CRTs are not fully implemented as intended (see Module 1.2). The EBP programme³⁶ recommends drawing on a broad range of stakeholder views in which empirical evidence linking service content to outcome is limited. Therefore, in this qualitative study, the aim was to examine stakeholders’ experiences of CRTs and their views regarding best practice.

Methods of data collection

Semistructured interviews and focus groups were conducted with different groups of CRT stakeholders: service users (n = 41 individual interviews), carers (n = 20 interviews) and practitioners (n = 147, comprising CRT staff, managers and referrers; nine individual interviews and 26 focus groups were conducted). The referrers group included clinicians from community mental health services, inpatient wards, liaison psychiatry services and general practitioners, thus representing a range of common referrers to CRT services. Participants were drawn from 10 mental health catchment areas across England. In addition, key experts (n = 11), involved in the development of CRTs internationally, were interviewed in order to provide a broad perspective on the history and theoretical origins of CRTs and on their intended aims and characteristics.

Topic guides were developed in collaboration with project advisory groups of service users, carers and clinicians, representing CRT stakeholders, to make sure that priority issues and the concerns of each group regarding CRT practice were covered. The overall aim was to explore views from all perspectives about what constitutes best practice in CRTs. Aspects of CRT care identified by previous research as problematic were included. Interviews with service users and carers were conducted by trained peer researchers (service users and/or carers) whenever available; the rest were conducted by non-peer study researchers. Data from practitioners were collected via focus groups, when feasible, or via individual interviews if it proved too difficult to convene focus groups. All interviews were audio-recorded and transcribed.

Analysis

Data were analysed using thematic analysis⁶⁶ supported by NVivo 9 software (QSR International, Warrington, UK). Using this analytical strategy, aspects of CRT work were explored that were perceived to be important, successful and unsuccessful, using both inductive and deductive approaches. Owing to the large corpus of data, a staged approach was used. An initial basic set of themes was developed, based on a small subsample of transcripts, in order to capture broad areas and organise the data. This set of themes was progressively elaborated through group discussions and an iterative process of reading and coding further transcripts. Later stages of analysis were focused on developing a nuanced understanding of parts of the data corpus relevant to our aims. Peer researchers, as well as members of project advisory groups representing all stakeholder groups, were involved in all the stages of the analysis. These collective processes can enhance validity by promoting reflexivity⁶⁷ and ensuring that the perspectives of all stakeholder groups are considered in the analysis.⁶⁸

Key findings

There was a high level of agreement between stakeholder groups, with 11 features of CRT work identified as important across the groups. These were organised into three broad domains: (1) organisation of care, (2) the content of CRT work and (3) the role of CRTs within the acute and continuing care systems. For each feature, similarities and variations in stakeholders’ views on successful and unsuccessful aspects of current CRT practice and implementation were considered. Findings are summarised below.

1. Organisation of care:
   1. Ease of access and speed of response were identified as being among the most important aspects of initial CRT contact for service users and carers. The need for 24-hour, 7-day access was emphasised, although the resource challenges of this were acknowledged. All stakeholder groups recognised the benefits of self-referral for known clients. Accident and emergency departments were viewed as the least satisfactory referral pathway.
   2. Regularity, reliability and clarity were valued by stakeholders as these facilitate trust, emotional support and monitoring of risk and change. A few service users reported poor communication and infrequent visits from staff as the least helpful aspect of their care, whereas a small minority of service users and carers described large negative impacts when staff were not reliable.
   3. Flexibility for referrals, timing of visits, forms of support and duration of contact were advocated by all stakeholder groups, with service users particularly valuing flexible timing of visits and involvement in their planning. Implementation issues identified include resource limitations and difficulties with predicting workloads.
   4. Staff continuity was greatly valued by service users and carers as it can facilitate trust and relationship building, whereas the lack of it was one of the least helpful aspects of CRT contact. Practitioners and CRT developers were aware of the importance of staff continuity and the need to improve it. Several strategies to prioritise continuity within a shift-working system were discussed.
   5. Staff mix and experience – service users and carers valued staff experienced in crisis work and CRT-specific staff training was viewed as essential by practitioners. CRT staff and developers identified multidisciplinary staff mix as one of the most important aspects of good practice. However, only a minority of CRT staff groups described their own teams as multidisciplinary. Practitioners often described overall staffing levels as stretched or inadequate because of resource limitations.
2. The content of CRT work:
   1. Involving the whole family – CRT developers and practitioners described family involvement at initial assessments as valuable in helping them to develop a holistic view of the crisis and to decide on the suitability of home treatment and CRT interventions. Although family involvement was valued by carers, many often felt excluded from decisions about treatment and support.
   2. Emotional support was extremely important for service users and many described this as the most helpful aspect of their CRT contact. Staff with excellent basic emotional skills, such as kindness and empathy, and staffing continuity, with opportunities to develop therapeutic relationships, were identified as facilitating provision of high-quality emotional support. A few service users and carers said that they had not received any emotional support, as their contacts with CRT staff were brief and focused on organising care or medication.
   3. CRT interventions – medication supervision was identified as the principal intervention, although stakeholders agreed that CRTs were often too narrowly focused on this at the expense of other forms of support. Practical interventions were valued, but their availability was often limited by time and resources. Psychological input, self-management and physical health checks were also valued but not frequently provided.
3. The role of CRTs within the acute and continuing care systems:
   1. Gatekeeping hospital admissions was seen as the most important function of CRTs by developers, with CRT capacity to prevent admissions under threat when this did not take place.
   2. Providing home-based treatment was valued across stakeholders for preserving freedom, social contacts and daily routines, and allowing greater privacy and safety for service users. Stakeholders from all groups were concerned about whether or not home treatment decisions were based on clinical need or a requirement to reduce hospital admissions.
   3. Continuity and communication with other services around discharge were seen as particularly important by stakeholders. However, all stakeholder groups reported problems in continuity of care and effective interservice communication. Almost all practitioners and developers reported misunderstandings and lack of knowledge about CRTs among referrers. A number of strategies were suggested to help improve working relationships and continuity with other services.

Strengths and limitations

To our knowledge, this was the first study to provide an in-depth, multiperspective qualitative exploration of the successes and failures of CRTs across a wide range of contexts, covering trusts in urban, suburban and rural areas of England. The perspectives and experiences of all relevant stakeholder groups were explored and compared. The large number of qualitative data and the diverse demographic, clinical, service use and professional characteristics of the study’s sample help ensure that the common themes identified generally reflect the views of these stakeholder groups.

Given that service users were recruited through CRT clinicians, it is likely that the study’s sample under-represents service users who were not very engaged with services or who had generally negative experiences of CRTs. Because of the challenges of reporting concisely on such a large data corpus, the study focused on common themes and prevailing views within each stakeholder group. Thus, some less common concerns or views have not been thoroughly explored in depth, and the approach in this initial analysis of a very large number of interviews is necessarily broad rather than in-depth.

Recommendations for future research

Many of the findings from this study had direct implications for the development of a more highly specified CRT model, especially in relation to improving service user and carer experiences. Priority areas for service improvement and for further research in optimising team function and outcomes include continuity of care, provision of emotional and practical support, carer involvement, and the quantity and quality of contact with CRT staff. Findings also highlight challenges to CRT implementation in contexts of stringent resource limitations and complex service configurations. Most previous UK CRT research has been conducted in urban areas; some stakeholders in our interviews raised challenges to implementing the CRT model fully in more rural areas (e.g. in achieving a rapid response service and providing frequent home visits). Further research is needed to understand and assess barriers of, and facilitators to, successful implementation of good CRT care in different contexts. In relation to initiatives for CRT improvement, qualitative research including multiple stakeholder perspectives is a rich source for understanding the impact of initiatives and barriers of, and facilitators to, achieving intended improvements. Use of implementation science methodologies in understanding how to overcome barriers and achieve change is a priority for future research.

Aims

Fidelity scales are measures of the extent to which services adhere to a model of high-quality care, based on evidence and on stakeholder consensus. They are designed as tools to support quality improvement.⁵⁶ There is evidence for some fidelity scales of an association between higher fidelity and superior service outcomes.⁷⁰ Fidelity scales have been developed for use in some mental health service models,⁷¹ especially in the USA, but none exist for CRTs.

The aim of this study was to develop a highly defined and validated fidelity scale for CRTs. The study aimed to (1) systemically develop a CRT fidelity scale, (2) test its feasibility and utility in practice and (3) conduct an initial investigation of its psychometric properties.

Methods

Concept mapping⁷² was used to construct the CRT fidelity scale. This is a method used in the development of measures to identify and group into relevant domains a set of priority items based on stakeholder consensus. Potential service model characteristics were identified from module 1 (the literature review),⁵⁸ the national survey of CRT managers⁶³ and a qualitative investigation of CRT stakeholder perspectives. All yielded potential items for inclusion in a longlist of characteristics of high-quality CRT delivery. Next, this longlist of characteristics, identified as associated with good practice in CRTs, was condensed and combined into a list of fidelity scale statements by a group of academic, service user and clinician stakeholders by, for example, removing duplicates and combining closely related items. Following this, a series of concept mapping meetings involving stakeholders with different perspectives was held. Participants were asked to prioritise statements in terms of their importance for CRT best practice and to sort statements into groups of conceptually related statements. Specialist Ariadne concept mapping software (version 1.0; Talcott, Amsterdam, the Netherlands) was then used to analyse participants’ grouping of statements, using a form of cluster analysis, to generate a series of ‘cluster solutions’, which distinguished domains of CRT fidelity to reflect how participants grouped statements. The software also generated a mean importance rating for each statement from participants’ prioritisation of statements, which allowed the fidelity statements considered most important by stakeholders within each conceptual domain to be identified. Cluster solutions generated by the software were then reviewed by a small group of stakeholders, who selected the cluster solution that produced the most conceptually coherent set of domains of CRT fidelity, and named each domain, reaching consensus through discussion. The chosen cluster solution, that is, the final concept map, was then used to develop the CRT fidelity scale. Statements representing each fidelity domain were selected for inclusion in the fidelity scale. Statements of higher importance within each domain were selected for inclusion. Operational definitions were developed for each statement and anchor points specified for scoring, with reference to the development work carried out in module 1. Adherence to fidelity items is measured on a five-point scale. In common with other fidelity scales,⁷⁰ a score of 5 on each item was taken to represent excellent model fidelity, that is, a service was adherent to best practice standards for this item. A score of 3 was taken to represent moderate fidelity, that is, a service was adhering to some essential requirements for best practice in this area, but not achieving all desirable elements of best practice for this item. The measure was thus able to distinguish services with ‘good’ overall model fidelity (i.e. a mean score of ≥ 4 per item) and services with ‘fair’ model fidelity (i.e. a mean score of 3–4 per item). At each stage of development, further stakeholder consultation was sought.

A fidelity review process was developed and piloted for collection of fidelity scale data. Reviews took place over 1 full day and were carried out by three reviewers: one clinician, one service user/carer and one researcher. CRT managers were contacted before the review to identify required documents and to establish the schedule. The review day involved interviews with managers, staff, liaising service staff, service users and carers; reviewing 10 anonymised case records; and reviewing service policies and records. Reviewers collected data using checklists and interview schedules, and scored fidelity items together at the end of the review. A draft fidelity review report was sent to team managers for clarifications before a final report and fidelity score were provided. Pilot reviews were initially conducted in four CRTs, followed by a 75-team review with CRTs from different parts of England, Scotland and Wales (see Crisis resolution team fidelity survey). Following reviews, researchers sought feedback from managers regarding the acceptability of the fidelity review and the validity and clarity of the fidelity scale.

Three psychometric properties of the fidelity scale were investigated:

1. Face validity was assessed through feedback from managers of CRTs participating in the pilot.
2. The presence of floor or ceiling effects and the ability of items to discriminate between higher and lower fidelity teams were investigated by assessing the range of scores from the 75-team review.
3. Inter-rater reliability was investigated using an extended vignette designed by researchers using mock fidelity review records.

Results

To construct the scale, an initial list of 232 identified service model characteristics was condensed to 72 fidelity scale statements by examining them for duplication and overlap. Sixty-eight participants were then involved in concept mapping, including representatives from all stakeholder groups (including service users, carers, CRT clinicians and managers, and CRT researchers). A four-cluster solution was chosen as the most coherent concept map. The clusters were named as (1) referrals and access, (2) content and delivery of care, (3) staffing and team procedures and (4) location and timing of care. Thirty-nine statements receiving a high level of endorsement were included in the final fidelity scale, representing each of the four clusters.

The three broad stakeholder groups in the process were service users and carers, CRT clinicians and others (including CRT researchers). All statements that were rated as very important (i.e. a mean score of > 4) by any of the three groups or as moderately important (i.e. a mean score of > 2.5) by more than one stakeholder group were included in the scale, either as a distinct item or among item-scoring criteria. Table 4 summarises the four clusters of items (subscales) defined through the process and the mean importance score for the items in each.

TABLE 4

The relationship of the CRT fidelity scale structure to concept map clusters

The first version of the fidelity scale was piloted in 75 teams in a range of urban and rural locations and catchment sizes. Feedback indicated that fidelity reviews and reports received afterwards were perceived as acceptable and helpful, although reviews were time-consuming. The first version of the fidelity scale was reviewed after 50 reviews. All 39 items were retained, with minor changes made to 19 items in creating version 2 of the fidelity scale. Version 2 was piloted alongside the original version in nine CRTs.

Seventeen reviewers participated in the inter-rater reliability exercise, using version 2 of the fidelity scale. An estimated kappa correlation of 0.65 [95% confidence interval (CI) 0.54 to 0.76] between ratings was identified, indicating moderately high similarity between ratings within an item. The estimated intraclass correlation (ICC) averaged over the 16 raters was very high, at a value of 0.97 (95% CI 0.95 to 0.98). Consistency of agreement ICCs produced nearly identical values (i.e. an ICC of 0.97), equivalent to Cronbach’s alpha.

Results indicate that the fidelity scale distinguished between higher- and lower-fidelity CRTs, with a wide range of scores obtained on each item. Piloting version 2 of the scale generated modest changes to fidelity scores compared with version 1 piloting, but increased scale clarity.

Strengths and limitations

Using concept mapping, the study systematically developed the first CRT fidelity scale, based on best available evidence and stakeholder input. The so-far inconsistent implementation of CRTs has limited the potential benefits of providing this service model.²⁴ The CORE fidelity scale has potential to support CRT model implementation and aid quality improvement work by providing a clear specification of this service model. This addresses the study’s overarching aim of improving the care provided to service users experiencing an acute mental health crisis.

There were two limitations of the study’s CRT fidelity scale development: the scope of stakeholder consultation and the degree of testing of psychometric properties conducted.

It was aimed to include stakeholder views at all stages of development. However, during concept mapping, clinicians outnumbered service users and carers in terms of input, and some stakeholder groups (e.g. primary care referrers to CRTs and emergency services) were not represented. Stakeholder input was largely from the UK, so it is not known how far the fidelity scale represents views of international stakeholders and how appropriate it is for use outside the UK.

Research implications

The content and criterion validity of the study’s fidelity scale require investigation to demonstrate the legitimacy of the scale as a measure of service quality, and further testing of the scale’s inter-rater and test–retest reliability is also a priority. A clinical need to develop and evaluate a resource manual to help CRTs achieve service standards and aid quality improvement³⁸ has been addressed in the (module 3) CORE SIP trial.⁷³ There is also a need to explore the feasibility and utility of the fidelity scale in international contexts where the model is used, such as in Norway and the Netherlands.

The development of the CRT fidelity scale has demonstrated concept mapping to be a feasible method in developing and defining the CRT service model. The development and piloting of the study’s CRT fidelity scale provides a model of PPI in research and audit.⁷⁴

Crisis resolution team fidelity survey

As well as serving to pilot the CORE CRT fidelity scale, the 75-team survey provided useful information about implementation of the CRT model nationally in England. See Appendix 4 for a reference to the published results of this work.⁶⁹

Forty-seven NHS trusts were approached, with 149 CRTs within their catchment areas. Of these, 75 CRTs from 27 trusts took part in the survey. The study sample of 75 teams comprised 70 teams in England, one team in Scotland and four teams in Wales.

In the 75 teams surveyed, the total scores ranged from 73 to 151 (possible range 39–195). The median score was 122 and the interquartile range (IQR) was 111–132 (these two findings were not reported in the published paper⁶⁹ on the development of the scale; see Appendix 4). Higher scores indicate higher model fidelity.

The median scores per item across all 75 teams for each subscale within the CRT fidelity scale are provided in Table 5. The low score for the ‘location and timing of help’ subscale reflects the fact that many CRT teams had no access to residential crisis houses or acute day units, and typically provide a less intensive service (i.e. less frequent visits to service users) than recommended in the CORE scale and in the original CRT guidance.

TABLE 5

Median scores on CRT fidelity scale subscales (n = 75 CRTs)

The fidelity items for which teams typically showed poor fidelity to the CORE CRT model (median score of 1 or 2) and the items for which teams typically showed good model fidelity (median score of 4 or 5) are presented in Table 6.

TABLE 6

Areas of high and low fidelity in the CORE CRT fidelity survey (n = 75 CRTs)

There are three main findings from this survey regarding implementation of the CRT model in practice in England. First, CRTs typically achieve only moderate fidelity in adhering to an optimal CRT model: teams’ scores ranged from low to moderately high fidelity; no team achieved an average item score of > 4, which would indicate very good model fidelity. Second, the range of scores achieved for each item suggests that the CRT fidelity scale is not unrealistically rigorous: all items were being delivered with high fidelity in some CRTs and, therefore, are achievable in routine care. Third, although areas of strength and limitations regarding CRT model adherence varied from team to team, the study’s survey shows that CRT teams typically struggled to provide core elements of the CRT model specified in government guidance,³ such as rapid response, a range of care from a multidisciplinary team and help with planning recovery and future crisis response.

This study thus corroborates the findings of previous surveys that relied on CRT managers’ reports of their teams’ organisation and service delivery,^24,63 and suggests that the CRT model has been inconsistently and only partially implemented in England. It suggests that a national mandate and policy guidance may in themselves be insufficient to ensure implementation of a complex service model as planned. It reinforces the need to develop effective resources to support implementation and service improvement in CRTs.

Aims

A cluster randomised trial was conducted in order to evaluate the impact of the CORE CRT SIP. CRTs allocated to the intervention arm received the SIP over a 1-year period; control CRTs did not receive the intervention. The trial tested the hypotheses that a CRT SIP can improve service users’ experiences of CRT care (primary hypothesis), reduce acute service use and improve CRT staff well-being. The trial also aimed to investigate whether or not fidelity scores increase in CRTs receiving the intervention, and to explore the associations between CRT fidelity scores and service outcomes. Through a qualitative evaluation, the trial sought to understand stakeholders’ experience of the SIP and barriers to, and facilitators of, its implementation.

Methods of data collection

Twenty-five CRTs were recruited from eight NHS trusts across southern England and the Midlands, covering urban, suburban and rural areas. CRTs identified by the earlier CORE fidelity survey as already achieving good model fidelity were excluded, as were CRTs anticipating any major service reorganisation during the trial period. At least two participating CRTs from each NHS trust were required, to allow randomisation of CRTs within trusts. Fifteen teams were randomised to receive the SIP, which involved support from a CRT facilitator, access to online implementation resources and feedback from regular service audits (for further information regarding the intervention, see Module 2.2). Ten teams were allocated to the control group. The primary outcome was service users’ satisfaction with CRT service, as measured by the Client Satisfaction Questionnaire (CSQ).⁷⁹

Service user satisfaction was chosen as the primary outcome because of the evidence from previous qualitative studies and surveys that the implementation of CRTs has not clearly improved service users’ experiences of mental health crisis care to the extent hoped for, and which trials indicate is possible.⁵⁸ The specific choice of the CSQ as the primary outcome measure was because it was brief and widely used, and has proved sensitive to differences between service models in other studies in acute care settings.⁷⁹ The calculated sample size for the trial provided 95% power to detect half of 1 standard deviation (SD) difference between groups in mean satisfaction measured by the CSQ (3.5 points assuming a typical SD of 7.0 points) using a two-sided test, allowing for within-team clustering (ICC 0.05 for clusters of size 15) and unequal 3 : 2 (intervention-to-control) randomisation.

Randomisation was stratified by NHS trust to ensure that within each trust some CRTs received the intervention, and some acted as control teams. A statistician, independent of the study, generated allocation sequence lists and conducted randomisations for all participating teams within each NHS trust at the same time, once baseline fidelity reviews had been completed at all sites within the trust. There were no blinding arrangements, as study researchers and CRT staff were aware of teams’ allocation status. Service user participants providing data for the trial’s primary outcome and trusts’ informatics teams providing service use data are unlikely to have known whether or not teams were receiving the SIP.

The trial comprised the following components:

• Quantitative evaluations of service user experience – from each CRT, the trial aimed to sequentially recruit 15 eligible service users at baseline and another sample of 15 service users at follow-up (months 10–12 of the trial intervention period), providing a total sample of 375 at each time point. CRT staff were asked to screen then contact all eligible CRT service users during this period to ask about their willingness to speak to a researcher, and researchers then sought to contact all potential participants until the target of 15 per team was achieved. Eligibility criteria were that participants had used the CRT service for at least 7 days, were able to read and understand English, had the capacity to provide informed consent and did not pose too high a risk to others to participate. At each time point, researchers administered two structured questionnaires (i.e. the CSQ⁷⁹ and Continu-um⁸⁰), which measured perceived continuity of care.
• Quantitative evaluations of staff well-being – all clinical CRT staff were invited to complete online self-report questionnaires on staff morale (as measured by the Maslach Burnout Inventory⁸¹), job satisfaction (as measured by the Work Engagement Scale⁸²) and psychological health (as measured by the General Health Questionnaire⁸³) at baseline and follow-up (months 10–12 of the trial intervention period).
• Quantitative evaluations of service use – anonymised demographic and service use data were collected from electronic patient records at baseline and follow-up. The data were collected for two cohorts: service users admitted to CRT care during a 1-month period ending 6 months prior to the study start date (baseline) and service users admitted in month 7 of the study intervention year (follow-up). These data were used to assess rates of re-admission to acute services following CRT care. All service users admitted to acute services over a 6-month period up to the study baseline date and during months 7–12 of the intervention year; these data were used to assess total hospital admission rates in the catchment area of each CRT. Study researchers provided a pro forma to appropriate administrators or informatics team specifying what type of data needed to be collected and for which time period. The study outcomes were then calculated from this raw data. Data were collected only for patients aged 18–64 years, that is, the age range included by all CRTs to ensure comparability of data across sites. Details of how patient records data were defined and operationalised are provided in the published results paper⁸⁴ for the workstream 1 trial (see Appendix 6).
• Participating teams’ CRT model fidelity was assessed at the 1-day ‘fidelity review’ audit, to be conducted at baseline and at the 1-year follow-up using the CORE CRT fidelity scale.⁶⁹
• Qualitative evaluation – the CRT service improvement facilitators were interviewed each month throughout the intervention year in order to record implementation activities the teams had undertaken. Six case study sites were purposively selected following the 12-month fidelity reviews. At each study site, individual interviews were conducted with CRT managers, CRT facilitators and their clinical supervisor, and a focus group with CRT staff in order to explore their experience of the SIP.

Teams were recruited to the study between September and December 2014, with a 1-year trial intervention period. All baseline fidelity reviews were conducted before teams were randomised. Baseline data from service users and staff were collected following each team’s fidelity review; for some teams, baseline data collection was completed after randomisation.

Analysis

Statistical analyses were conducted using Stata^® (StataCorp LP, College Station, TX, USA) and qualitative data were analysed using NVivo 9 software. For each of the trial components, the analysis was as follows:

• Service user experience – in order to test whether or not participants’ satisfaction with the CRT was greater in the teams that received the intervention than in the control teams (our primary hypothesis), linear random-effects modelling was used with a random effect for CRT, controlling for mean baseline client satisfaction score by CRT. Perceived continuity of care was also compared between control and intervention teams using a similar linear random-effects model. ICCs were calculated for both outcomes, with CRT as the clustering variable.
• Staff well-being linear random-effects models were used to compare mean staff psychological well-being, mean staff involvement and mean staff burnout scores between the intervention group and control arm. ICCs are shown for all outcomes using CRT as the clustering variable.
• Service use – routine data were used on admissions to acute services, bed use and population size to compare service use patterns between intervention and control teams. Whether or not there were any significant differences between intervention and control teams in rates of re-admissions within 6 months of an initial admission to acute care were investigated. Poisson random-effects modelling was used with bed use, admission rate or rate of re-admission to acute care set as the exposure variable, as appropriate, and using trust as the random effect. Two analyses were conducted for inpatient admission and bed use outcomes. The first analysis compared intervention and control group teams at follow-up, adjusting for baseline score on the outcome measure. This provides the more valid comparison of change in outcome during the study period. The second analysis adjusted for catchment area population, thus providing an alternative absolute comparison of inpatient admissions and bed use in the two groups. ICCs are shown using trust as the clustering variable.
• CRT model fidelity – the relationship at team level was explored between the change in team fidelity scores from baseline to the 1-year follow-up and changes in five study outcomes: service user satisfaction, staff job satisfaction, inpatient admissions, inpatient bed use and re-admission following CRT care. The relationship between fidelity change and outcomes was analysed using linear regression.
• Qualitative evaluation – descriptive data were reported on implementation activities and service initiatives that took place during the study intervention year. The time spent by staff on the online resource kit was captured by Google Analytics (Google Inc., Mountain View, CA, USA). Qualitative data from interviews and focus groups on barriers to, and facilitators of, achieving change, based on the SIP, were analysed using thematic analysis.⁶⁶

Results

All 25 CRTs were successfully recruited and retained in the trial. Sixty-two per cent of the eligible service users, who expressed willingness to CRT staff to participate in the study, gave informed consent to researchers to complete measures of satisfaction and continuity (353/567 at baseline and 371/594 at follow-up). The mean age of service user participants was 43 years (SD 15 years) at baseline and 42 years (SD 14 years) at follow-up; the majority were female (59% at baseline and 61% at follow-up) and most identified as white (86% at baseline and 81% at follow-up). At each time point, 79% of clinical staff then working in the participating CRTs completed the survey (441/560 at baseline and 432/544 at follow-up). The mean age of staff participants was 43 years (SD 10 years) at baseline and follow-up; the majority identified as female (66% at baseline and 63% at follow-up) and white (68% at baseline and 65% at follow-up). (For further information about recruitment rates and service user and staff participants’ characteristics, see the paper⁸⁴ referred to in Appendix 6.) No serious adverse events (SAEs) were identified during participant recruitment and data collection, and no study-related harms were reported by participating CRTs during the trial intervention.

The key findings are summarised in Table 7. There was no statistically significant difference in service user satisfaction (primary outcome) or service user-rated continuity of care between the intervention and control group teams. There were no significant differences between the two groups in three of the four staff self-rated well-being measures; better general psychological health outcomes for intervention group staff just achieved significance. There were significantly fewer inpatient admissions and inpatient bed-days (including both voluntary and compulsory admissions) at follow-up in the intervention group than in the control group, with adjustment for baseline rates. There were no significant differences between the two groups in the rates of compulsory admissions or rates of re-admission to acute care for people who had already experienced an episode of CRT care.

TABLE 7

Results for service user, staff and service use outcomes at follow-up

There was a weak correlation between an increase in team fidelity scores over the intervention period and improved patient satisfaction, with a correlation coefficient of 0.34 (p = 0.10). This can be interpreted so that a change of 10 fidelity points corresponds to a change of 0.65 CSQ points. There was a weak correlation between an increase in fidelity scores and a reduction in re-admissions following CRT care (correlation coefficient 0.38; p = 0.06). No significant association was found between the change in fidelity scores and changes in hospital admission rates, inpatient bed-days or staff well-being. For a breakdown of CRTs’ fidelity scores and relationships with outcomes, see the paper⁸⁴ referred to in Appendix 6.

The qualitative evaluation established three broad domains capturing stakeholder views and experiences of the SIP (for a more detailed description, see Appendix 7):

1. Components of the SIP – stakeholders described support from the CORE team as well co-ordinated, with regular, ongoing help provided when needed. Some teams reported finding their facilitator easily contactable and approachable, whereas others felt that facilitators lacked presence within teams. Most teams and facilitators felt the role was more helpful when facilitators were internal and known to the team. Experiences of the scoping day were largely positive, and stakeholders felt that it promoted engagement and action planning. Fidelity reviews were seen as time-consuming, but some teams found them helpful in motivating staff and helping them reflect on their practice. Some stakeholders did not see the review as accurately measuring a team’s practice. The fidelity review report was considered valuable feedback, helping teams with quality improvement planning. The resource pack website was somewhat underutilised and was mainly accessed for browsing and fact-finding.
2. Process factors – lack of time to engage with the programme was the most frequently cited process factor, with the busy and overstretched nature of CRT work emphasised by all stakeholder groups. Several stakeholders stated that 12 months was not enough time for teams to implement service changes and see their effects. Staff motivation and morale was the second most frequently cited process factor, with the potential to influence programme implementation positively and negatively. The majority of stakeholders mentioned team ownership as an important factor affecting input and engagement with the study. The programme was described as increasing team ownership by increasing awareness of how staff should be practising. Many stakeholders viewed the programme as a structure to support service improvement and as a means of benchmarking their practice. Support from seniors in the trust was considered important as it could increase motivation and engagement. Service user involvement was also highlighted, although it was not achieved in all teams.
3. Impact of the programme – the majority of stakeholders described service innovations resulting from participation in the programme. Stakeholders also reported various ways in which service user experience improved following the programme. Team communication was seen as being improved, as the programme promoted a shared sense of purpose and understanding of the team’s tasks. Team purpose was enhanced as stakeholders were encouraged to address areas of practice that may previously have been neglected. Staff motivation was perceived as having improved at both individual clinician and team levels. Many stakeholders wished for quality improvement work to continue after the study end, seeing CORE as part of a bigger whole. However, two teams described a lack of impact of the programme. Lack of resources and staff turnover were cited as barriers to implementing SIPs.

Successes

Fairly good response rates of > 60% of service users referred by CRTs and nearly 80% of CRT staff were achieved. The primary outcome was negative, but our results did suggest that the implementation of the CORE SIP may have led to a reduction in hospital admission rates and inpatient bed use, when adjusting for baseline scores. Given the importance of reducing hospital admissions as an aim of CRTs, and to patient experience, it was believed that this positive finding provides an encouraging indication that the CORE CRT SIP is a worthwhile intervention. Although no significant differences were found in other outcomes, there were non-significant trends in favour of the intervention group teams regarding patient satisfaction, re-admission to acute services following CRT care and all staff well-being outcomes. The majority of intervention teams improved model fidelity over the study period, whereas none of the control teams did, which suggests that our programme was successful to some extent. There was also some evidence across all teams of a small positive association between increase in fidelity and improvement in outcomes for patient satisfaction and re-admission following CRT care.

The inclusion of 25 CRTs in varied geographical and sociodemographic contexts suggests that the trial findings may have general applicability to CRTs in England; however, generalisability to other national contexts is uncertain.

The qualitative interviews show that, overall, stakeholders had positive experiences of the SIP. The majority of stakeholders were able to describe ways in which the programme had a positive impact in their team, resulting in service innovations and improved staff motivation and service user experience.

Limitations and challenges

Several limitations arose from the study design. CRT service delivery and outcomes may have been affected by local policy directives and service initiatives that were independent of the trial intervention; the extent of these, and whether or not some teams were affected more than others, is unclear. As stakeholders pointed out, the 1-year follow-up period may have been too short for teams to implement their service improvement plans and to capture all the effects of the intervention.

Two limitations regarding the data collected for the primary outcome of service user satisfaction create uncertainty about whether the lack of a positive result on the primary outcome reflects a real lack of impact from the SIP on patient experience or methodological limitations of the study. First, the ICC for the primary analysis (ICC 0.12) was somewhat larger than anticipated (i.e. ICC 0.05), suggesting that the power of the main study was lower than planned (83%). However, the observed reduction in CSQ of 0.97 units was also far lower than the anticipated reduction of 3.5 units, and too small to be very meaningful in terms of patient experience, so this reduced power is unlikely to have substantially influenced the study findings. Second, and more importantly, the study participants may have been unrepresentative. It was not possible to establish from CRT teams how many service users were eligible and how many were approached by staff about taking part in the survey of satisfaction to measure the primary outcome for the study. Therefore, it cannot be determined whether recruited participants were representative of the CRTs’ service users in general or whether they over-represent those who are best engaged with CRT staff. This might also relate to the ceiling effect that was observed with the trial primary outcome measure (i.e. the CSQ). Service user participants were highly satisfied with care compared with previous studies.^15,18 In addition, the brief study measure might have not been sensitive enough to capture differences in service user satisfaction with CRTs.

Neither CRT staff nor fidelity reviewers were blind to teams’ allocation status during follow-up CRT fidelity reviews. Teams in the intervention group might have been more motivated to demonstrate fidelity gains than those in the control group. Reviewers may have also been subconsciously biased in favour of the intervention group. These phenomena may have served to exaggerate the difference found between intervention and control teams’ fidelity scores.

Study researchers were unable to independently verify that all relevant records were provided from NHS patient records, as they were anonymised. One trust was unable to provide information about whether inpatient admissions were compulsory or voluntary at follow-up. Researchers could not determine from the information provided whether service users were physically present on the ward during the whole of their inpatient admissions or whether they were on leave.

Recommendations for future research

Outcomes from the SIP that were trialled showed some promise in relation to effects on inpatient admission, but the primary study hypothesis was not confirmed. This suggests that the SIP and the fidelity scale are a potential basis for quality improvement in CRTs, but that further refinement is still needed.

Only a weak relationship was found between changes in overall fidelity scores and changes in outcome, and not for all key CRT outcomes. Therefore, the criterion validity of the fidelity scale and its utility as a marker of CRT service quality should be further explored. More robust testing of inter-rater reliability of the scale is needed in vivo.

It is possible that the critical components of CRT services lie within the CORE fidelity scale, but are not sufficiently weighted for teams’ overall fidelity scores to relate closely to service outcomes. Future research could further investigate the relationship between team outcomes and specific fidelity scale item or subscale scores.

A future economic evaluation could explore the cost-effectiveness of the intervention as delivered in this trial. A process evaluation could helpfully examine which aspects of service provision were addressed in service improvement plans and where additional support may have been needed to enhance the impact of the programme. The types of activity undertaken by CRT facilitators to support teams’ service improvement could be explored from facilitators’ monthly reports to help identify any gaps in support.

Aims

The study aimed to review the evidence of short-, medium- and long-term clinical effectiveness of self-management programmes for SMIs through a systematic meta-analytic review compared with other comparable therapies or standard care. Quantitative synthesis of results from different studies was also conducted.

Methods of data collection

Studies were included if participants were adults, aged ≥ 18 years and diagnosed with a SMI, that is, with a clinical diagnosis of schizophrenia spectrum disorders (schizoaffective disorder, delusional disorder and psychosis), bipolar disorder or major depression. Studies with mixed populations (including personality disorder) using secondary care mental health services were also included. Participants at any stage of their illness, that is, as part of early intervention or more chronic/later stages, were included in this review. Further details and full search terms are provided in the paper¹⁰¹ referred to in Appendix 8; searches were conducted without time limit or geographical restrictions.

The types of self-management programmes included were those that aimed to help service users develop their own mental health self-management strategies and coping skills in order to achieve better clinical and/or social outcomes. The outcomes of the included studies were grouped as symptom-focused outcomes, service use outcomes, functioning outcomes and recovery-related outcomes.

Characteristics of the study design, the intervention, participants and outcomes for all available data at all provided time points were extracted. Authors were contacted and asked to provide any missing data.

Analysis

Studies were reviewed for bias using the Cochrane risk-of-bias tool,¹⁰² which was used to assess the quality of evidence by evaluating the risk of bias, inconsistency, indirectness, imprecision and publication bias. Extracted quantitative data were meta-analysed using appropriate random-effects models for dichotomous data (relative risk and 95% CIs) and continuous data [standardised mean differences (SMD)]. Outcomes were analysed at the end of treatment, post intervention (short term), up to 1 year post intervention (medium term) and over and including 1 year post intervention (long term). When more than one self-management intervention was evaluated in a study, these arms were combined for comparison with the control group. Review Manager software (RevMan 5.2; The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen, Denmark) was used to conduct the meta-analyses. When outcome data were reported for more than one follow-up point, the time point closest to 1 year post intervention was used.

Key findings

Thirty-two RCTs with 4961 randomised individuals met the eligibility criteria for this review. The types of interventions included typically offered guided self-management, supported by a clinician or peer worker. There was wide variation in the duration of programmes, and in the number, duration and frequency of sessions within programmes. The majority of studies were conducted in Western countries, with a smaller but significant proportion in Asian and developing countries (κ = 7). The majority of studies included participants who were currently living in the community, with eight studies recruited from inpatient settings. In relation to clinical diagnosis, 16 studies included participants with only schizophrenia spectrum disorder and seven included only those with a diagnosis of bipolar disorder. The remaining nine included mixed populations of participants with schizophrenia, psychosis, bipolar, major depressive disorder and personality disorder.

The meta-analysis found some evidence of short- and long-term positive impacts, albeit relatively small ones, of self-management programmes on total symptoms (SMD experimental vs. control at follow-up: –0.81, 95% CI –1.18 to –0.44; p < 0.00001), social functioning (SMD –0.55, 95% CI –0.92 to 0.17; p < 0.0000), self-rated recovery (SMD –0.24, 95% CI –0.36 to –0.11; p = 0.0003) and quality of life (SMD –0.26, 95% CI –0.39 to –0.13; p < 0.0001). Initial analyses found no clear effect on number of relapses or length of time in hospital; although heterogeneity was high in these meta-analyses, numbers were small and there was some evidence of an effect on relapse, with one study moved to reduce heterogeneity.

Strengths and limitations

Using a broad inclusion criterion for SMI in this review, and including studies of mixed populations of secondary mental health service users, has ensured that the findings of the review are relevant to the CRT context of the workstream 2 CORE study trial. All studies included in this review were RCTs, but there was variation in the reporting of sequence generation and allocation concealment and, as is common in complex interventions, the blinding of participants and personnel was not always consistent. The greatest cause for concern was the selective reporting of outcomes, which was noted or not clearly reported in over half of the studies reviewed. Furthermore, the small number of studies and small number of participants in the studies meant that it was difficult to discern any evident publication bias.

Other limitations were high heterogeneity in many of the meta-analyses conducted, and relatively infrequent inclusion of re-admission and other service use outcomes. Comparisons of self-management with treatment as usual and those with active controls were pooled together. This potentially makes interpretation of results more challenging, although, in fact, we found no evident differences in results relating to type of comparator. There were also variations in the timing of the follow-up comparison.

Recommendations for future research

The review does not determine the relative clinical effectiveness of different types of self-management programme; further research and an established typology of self-management programmes are needed to achieve this. Importantly, future research needs to be of higher quality. Additional design features should also be considered, for example collecting longer-term follow-up data and applying consistency to the spread of outcome measures, with relapse and service use outcomes consistently measured. Future studies should also examine whether or not any variations in delivery could have an impact on service user outcomes and costs. Establishing whether or not guided self-management programmes (e.g. supported by clinicians or peer workers) are more effective than self-guided programmes is clearly relevant to determining the most cost-effective types of intervention. There is also scope for researching the application of computer-based programmes or applications (‘apps’), as this is a growing area of health research because of the potential for wider reach and reduced costs.

Research aims

Comprehensive evidence has not previously been available about how effective peer-support programmes are and whether or not they improve outcomes for people with severe mental health problems. Although some systematic reviews of peer support are available, these are limited to subtypes of peer support or for people with depression only. This study aimed to review trials of community-based, peer-provided support for people with SMI.

Methods of data collection

The effects of peer-provided interventions were examined via objective outcomes, such as employment and re-admission to hospital and psychiatric symptoms, as well as self-reported outcomes, such as quality of life, recovery, empowerment, hope and satisfaction with services. Eligibility criteria were established. Published and unpublished RCTs, including cluster RCTs and factorial RCTs, were included.

Analysis

The SMD and Hedges’ g were calculated for continuous outcomes, and studies were weighted using the inverse of variance. For dichotomous outcomes, risk ratios were calculated and studies were combined using the Mantel–Haenszel method. All outcomes were reported with 95% CIs using random-effects models. Statistical heterogeneity was assessed by inspecting forest plots, conducting the chi-squared test and calculating the I²-statistic. Meta-analysis was conducted using RevMan, and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system was used to rate confidence in the evidence from each analysis of pooled data.

Key findings

Eighteen trials (including 13 from the USA), with a total of 5597 participants, were included. Programme content and participants’ characteristics varied substantially, and outcomes were incompletely reported. Independent reviewers found a high risk of bias. It was only possible to synthesise a small number of studies. The quantitative data synthesis showed some evidence that peer support was associated with positive effects on empowerment from mutual support (SMD –1.44, 95% CI –2.79 to –0.09), and on recovery (SMD –0.24, 95% CI –0.39 to –0.09) and hope (SMD –0.14, 95% CI –0.27 to –0.02) from peer-support interventions post treatment and at follow-up. However, these effects varied across different types of peer support and confidence and this evidence was graded very low. There was very little or no evidence to suggest that peer support was associated with positive effects on satisfaction with services, overall symptoms or hospitalisation.

Strengths and limitations

Through using a broad inclusion criterion for SMI in this review, and including studies of mixed populations of secondary mental health service users, it was ensured that the findings of the review are relevant to the CRT context of the workstream 2 CORE study trial. Searches were highly sensitive, and the review provides a comprehensive synthesis of outcomes that were considered important by triallists and service users. The main outcomes were identified a priori, although hope and empowerment were added post hoc. Only randomised trials were included, as they provide the best evidence of efficacy, in line with best practice guidelines for systematic reviews.¹⁰²

Nevertheless, the conclusions were limited by the small number of eligible studies, which did not allow the conduct of subgroup analyses. Another limitation was the substantial variation in programme content and trial populations, which may have contributed to observed heterogeneity. Confidence in existing evidence was low because of poor study design and incomplete reporting of included studies. Peer support was categorised based on previous typologies, as there were no clearly defined models or fidelity criteria. Overall, there was not enough current evidence to conclude that peer-support interventions are either effective or ineffective for people with severe mental health problems.

Recommendations for future research

This review highlighted the need for high-quality research to accurately evaluate the clinical effectiveness and cost-effectiveness of peer-support interventions for people with severe mental health problems. Further research is needed to develop, clearly describe and test theory-based interventions. It was difficult to distinguish different types of peer support using an existing typology as programmes varied in many factors. A distinction between structured self-management-focused programmes and longer-term, less structured ones could be considered in future typologies. Future research should develop and describe clear models of peer support in order to facilitate future trials and help future reviewers synthesise the data more effectively. Trials including a process evaluation could help understand whether or not the peer-support programme is implemented as intended and mechanisms by which any treatment effects are achieved.¹⁰⁴ Future research should also examine critical ingredients that might contribute to programme success and appropriate target populations.

Most of the studies included in this review were poorly reported and failed to adhere to the CONSORT guidelines for conducting and reporting trials.¹⁰⁵ This review further highlights the need to improve standards for reporting trials of social and psychological interventions. Future trials of peer-support programmes must be registered in advance,¹⁰⁶ publish their protocols, clearly state all outcomes and time points to be assessed, and describe interventions in full.

Research aims

Although self-management peer-support programmes have shown promising evidence of effectiveness, the empirical evidence for their effectiveness is still limited, as shown by the systematic reviews we conducted (see Module 4.1 and Module 4.2). Following on from these reviews, it was aimed to identify or develop a peer-supported, self-management programme appropriate for people discharged from CRT care.

Methods

The intervention was selected and developed through consultation with expert reference groups of CRT staff, as well as service users and carers. Multistage qualitative interview and focus group consultations were held to examine programme acceptability and feasibility and determine programme content. At each stage, consultations were conducted by a combination of service users, carers, clinicians and academic researchers. Focus groups and interviews were audio-recorded and then transcribed to be analysed.

Initial semistructured interviews explored service users’ views on peer support and priorities for support following CRT care. Semistructured interviews were conducted, usually by service user or carer researchers, with 41 service users recruited from CRTs in 10 NHS trusts in England. A self-management intervention (i.e. the Personal Recovery Plan)¹⁰⁸ was then identified by the CORE service user, carer and clinician advisory groups as the best potential fit for the study’s objectives and to address the priorities of interviewed peer supporters. The Personal Recovery Plan¹⁰⁸ structure was aligned to the themes reported in our systematic review findings (see Module 4). Focus groups of the key stakeholders were conducted to address the design of the proposed peer-facilitated self-management programme. Further modifications were identified through a feedback session with the newly recruited PSWs, as well as reviews by members of the study team, including CRT senior clinicians.

For modelling purposes, a pre-pilot of the intervention was facilitated by four trained PSWs, recruited from within the CORE service user advisory group, with 11 CRT service users post discharge. A psychologist supported weekly group supervision with the PSWs and was available if additional support was needed. Following completion of the one-on-one pre-pilot intervention, qualitative feedback was obtained through a focus group with the PSWs (n = 4) and individual interviews with the service users (n = 9). Both groups provided qualitative feedback on their experiences of the programme and suggestions about how it might be improved.

Further qualitative feedback was gathered from participants in the treatment arm of a pilot RCT embedded in the crisis resolution team service (see Module 5). Through these stages of consultations and feedback, a manualised peer-supported intervention for recovery and relapse prevention following a mental health crisis was developed and refined.

Analysis

Qualitative data from interviews were analysed by a team of researchers using thematic analysis. NVivo 9 software was used to code and analyse the data into different themes. The analytic strategy combined inductive and deductive approaches in order to explore initial research questions, as well as themes that characterised participants’ own experiences. A collaborative and iterative approach was adopted: a research team lead by a principal analyst read transcripts, developed a coding frame and regularly discussed emerging themes. This process was repeated at all qualitative phases. In addition, the feedback obtained from the stakeholders and pilot participants was later incorporated into the design of the intervention.

Results

Data from initial qualitative interviews and focus groups with CRT staff, service users and carers indicated support for the overall purpose and content of the programme, and for PSWs as providers of the intervention. Service users’ expressed needs were in line with the self-management approach. They endorsed sessions on relapse prevention, crisis planning, signposting and goal-setting as potentially useful. A small number of negative views, mainly concerns raised by carers and clinicians regarding peer support, highlighted the need for adequate PSW training, protocols, supervision and recruitment procedures.

Feedback regarding the concerns raised about the programme guided the development of training and procedures for the pre-pilot. Participant feedback after testing in the pre-pilot was largely positive regarding the facilitation of the programme by PSWs. Nearly all participants stated that they benefited from the peer-support experience. Some reported that the relationship itself was more useful than the structured section of self-management workbook, which was underutilised. The workbook was modified based on qualitative feedback from both participants and PSWs in order to improve usability. Minor changes to the language in the workbook were made in discussion with the PSWs and the study service user and carer expert reference groups. By replacing a definition of recovery in the original plan, space was created in the study version for participants to record their personal meaning of recovery, following discussion with their PSW. More blank space was included throughout the plan to allow participants space to include information, potentially using a range of media, relevant to their personal recovery plans. Moreover, additional training for PSWs was provided to facilitate workbook use.

Strengths and limitations

In some cases, the peer-support programme could not be adapted to accommodate service users’ views and requests during feasibility testing and piloting. A few participants wanted more than 10 sessions with the PSWs; however, this could not be changed because of funding and requirements for a consistently delivered trial intervention. One service user expressed a wish for a more closely ‘matched’ peer, but it was not feasible to provide this given the small PSW teams at each site.

Overall, however, the study has employed a process of systematically and iteratively obtaining stakeholder feedback at the different stages of intervention development. This has helped develop an intervention that matches on the needs and priorities of the target client group. The resulting trial intervention is clearly defined and manualised to enable consistent implementation and replicability. The different stages of consultations were important to strengthen the robustness and usability of our programme. The consultations allowed us to gauge stakeholder acceptability, but they were also useful in identifying gaps and areas of key importance when training PSWs and integrating the programme into a crisis team setting.

As Lovell et al.¹⁰⁹ pointed out, there are both advantages and disadvantages in using an approach that develops theory and models before conducting an exploratory trial. Developing complex interventions is a time-consuming and resource-heavy process, and incorporating and balancing the viewpoints of different stakeholder groups can be labour intensive. Nevertheless, this process is essential as it systematically incorporates the evidence base with current viewpoints while iteratively testing and refining the intervention design as it develops. The effectiveness of this peer-supported intervention in terms of its impact on service user outcomes and the economic impact remains to be piloted (module 5) and then evaluated in a full-scale definitive RCT (module 6).

Aims

The pilot trial was used to test out trial procedures regarding recruitment, randomisation, data collection and storage, the study outcome measures and analysis plans. Qualitative feedback from study participants and PSWs was also sought to identify any further changes needed to the self-management workbook or arrangements for training, supervising and supporting the PSWs. This would allow amendments to the protocol to be made, if necessary, before the main trial to optimise the intervention provided to participants and its evaluation.

Methods

Forty participants were recruited from one CRT in an inner-London NHS trust and randomised to a treatment or control group. Twenty-one participants in the treatment group were offered up to 10 sessions with a PSW who supported them in using a self-management workbook. Nineteen participants in the control group were sent the self-management workbook but no additional guidance or peer support to help them use it. Patient-reported outcome measures were collected at baseline and 4-month follow-up interviews. Data regarding service use, including the primary outcome of re-admission to acute care, were collected from electronic patient records for the year before trial entry and the 1 year of follow-up.

Brief qualitative interviews were conducted with participants in the treatment group at the end of the trial intervention to explore their experience of the programme of support. A focus group was conducted with the PSW providers of the intervention about their views on content and implementation of the programme.

The few changes made to the trial protocol between the module 5 pilot trial and the module 6 main trial are reported in Module 6, Methods. Otherwise, the trial methods were retained as described in the section below on the main trial (see Module 6) and in the paper¹¹⁰ referred to in Appendix 11.

Results

Discussion

The pilot trial demonstrated that recruiting and retaining participants in a trial and to the trial intervention were feasible, and confirmed the acceptability of the trial intervention.

From the experience of the pilot trial, the following changes were made to the trial procedures and outcome measures for the main trial:

• Participants were categorised by screening clinicians as having psychosis or bipolar disorder or not, with the aim of recruiting at least half of the trial participants with psychosis/bipolar disorder to ensure that they were broadly representative of CRT service users in general.
• Procedures were improved for informing clinical services that participants in the treatment group were ready to be allocated to a PSW, and for checking that this allocation had been done. (The trial data officer logged this information, which was checked by the trial manager.) This was to ensure that prompt access to a PSW was provided for participants in the treatment group.
• A measure of perceived continuity of care⁸⁰ was dropped following the pilot and replaced by briefer measures of loneliness,¹¹¹ social network¹¹² and neighbourhood social capital.¹¹³ These measures were thought to be more likely to change directly as a result of the trial intervention, while omitting the continuity-of-care measure avoided extending the length of the interview schedule unduly.
• A measure of the participant-rated recovery orientation of the PSW¹¹⁴ was added as an additional measure for participants in the treatment group, completed as a telephone interview by the non-blinded study data officer once a 4-month follow-up interview had been completed. This was part of a planned process evaluation for the trial.
• The PSWs’ training programme was refined in response to PSW feedback about challenges that were encountered. Changes resulted in more discussion and guidance in training on ways of using the recovery plan collaboratively, and managing role boundaries and endings of support.

Because primary outcome data were obtained for all participants and no major changes were made to the trial intervention, a decision was taken (with approval from the independent Trial Steering Committee, the Trial Data Monitoring and Ethics Committee and the funders) to incorporate the pilot trial into the main trial. This turned the pilot trial in to an internal pilot: for this reason, outcomes data from the pilot trial are not reported separately, but included in the module 6 main trial results.

Aims and objectives

The aim of the study was to test the peer-facilitated self-management intervention that was developed (see Module 4.3) to bridge the gap between crisis and continuing care, reducing relapse and promoting recovery. This trial compares the clinical effectiveness and cost-effectiveness of this intervention, delivered by trained PSWs to people just discharged from the CRT, with treatment as usual, supplemented by distribution of the recovery plan booklet by post without any specific support in engaging with it and using it.

The primary hypothesis was that service users receiving the experimental intervention would be less likely to relapse within 1 year of discharge from the CRT (indicated by re-admission to any mental health acute care service) than those receiving the control intervention.

The secondary hypothesis tested whether or not being in the experimental rather than the control condition is associated with a longer time to first admission to acute care and fewer days in acute care over 1 year. In addition, the study tested the hypotheses that at the 4- and 18-month follow-ups the experimental intervention would result in better perceived recovery, greater self-management skills, greater satisfaction with services, fewer symptoms, less loneliness and better social networks than the control intervention.

The study also evaluated the cost-effectiveness of the peer-facilitated self-management intervention compared with the treatment-as-usual intervention at 18 months.

Methods

This trial was a rater-blind RCT with two parallel arms (with an allocation ratio of 1 : 1) powered on the primary outcome, with adjustment for clustering by PSWs. Participants were identified from the caseload of CRTs in six NHS trusts (four in London, one in South East England and one in the South West of England).

Analysis

All analyses were on an intention-to-treat basis and were conducted using Stata.

For the primary outcome (re-admission to acute care during the study period), data were analysed using logistic regression with random intercepts, adjusted for clustering by PSW. Diagnosis (psychosis or bipolar disorder vs. no psychosis/bipolar disorder) and centre were entered into the model as fixed effects.

For secondary outcomes, linear- or zero-inflated negative binomial regression with random intercepts was utilised when appropriate, with PSW as the random effect, controlling for the baseline value of the outcome, condition and centre. Random-effects Poisson regression was planned to assess the total number of days spent in acute care and Cox regression frailty model for time to first re-admission. However, on seeing the structure of the data, zero-inflated negative binomial with robust standard errors was more appropriate, given that more than half of participants had not spent any days in acute care since baseline. For number of days in acute care, the centre was entered into the model as a fixed effect.

Supportive analyses using analogous statistical methods to the primary analyses were conducted on the primary outcome, adjusting for any significant baseline differences between randomised groups and the amount of improvement for both groups between baseline and follow-up. Predictors of missingness were also explored and controlled for in supportive analysis (see Report Supplementary Material 2).

The primary cost-effectiveness analysis is a complete-case analysis of the incremental cost per QALY gained of the peer-facilitated self-management intervention compared with treatment as usual. Costs include the cost of the peer-facilitated intervention including cost of training and supervision as well as unit costs applied to mental health service use collected from mental health trust files. Costs are from baseline until 12 months, adjusting for costs 12 months prior to baseline. QALYs between 12 and 18 months were discounted at a rate of 3.5%. All analyses include clustering by PSW. Cost-effectiveness acceptability curves (CEACs) and 95% CIs were calculated using bootstrapping. Further details of the methods and results of the trial economic analysis are provided in Appendix 13.

Results

From the 3288 service users screened, 1848 were assessed as eligible and a total of 441 participants were recruited, including participants recruited for the pilot trial. The CONSORT flow diagram (Figure 2) shows participant flow through the study.

FIGURE 2

The CONSORT flow diagram for study recruitment and retention.

Forty participants were recruited in the internal pilot trial between 14 May 2013 and 12 November 2013. The remaining 401 participants were recruited between 12 March 2014 and 3 July 2015. Follow-up interviews were arranged as soon as possible after 4 and 18 months from the date of recruitment. The final follow-up interview took place on 23 February 2017. Participants had a mean age of 40 years (SD 13 years), 40% of those who gave information about their gender were male (175/438) and 65% identified as being white (285/438). At the 4-month follow-up, 344 participants completed the interview assessment (completion rate 78%); 255 participants completed the 18-month interview assessment (completion rate 58%). For a breakdown of participant recruitment numbers and further details on participant characteristics (see Report Supplementary Material 1).

Findings on the primary outcome favoured the intervention group: 29% of intervention group participants were re-admitted to acute care within 1 year compared with 38% of control group participants [odds ratio (OR) 0.66, 95% CI 0.43 to 0.99; p = 0.0438] (Table 8). This persisted with planned adjustments.

TABLE 8

Outcomes in terms of the intervention

Regarding secondary outcomes, time to re-admission was longer in the intervention group than in the control group. However, the numbers of days in acute care were not significantly different. Initial descriptive analyses found an unexpected difference between groups in the number of days between randomisation and discharge from the index acute care admission. In view of this, a post hoc sensitivity analysis was conducted excluding these days and including only days that were part of acute care re-admissions. The intervention group participants had fewer acute care days than control group participants, but not at a statistically significant level [experimental vs. control group: median number of days in acute care 0 vs. 0 (IQR 0–10 vs. 0–22); mean number of days in acute care 13 vs. 19 (SD 31 vs. 40); incidence rate ratio (IRR) 0.90, 95% CI 0.66 to 1.23; p = 0.515]. There was no evidence of a difference between groups in the number of contacts with the community mental health teams, which were the main secondary care providers after CRT discharge.

At the 4-month follow-up, overall satisfaction with mental health care received was greater in the intervention group than in the control group. This was because there was a difference in mean satisfaction scores between the randomised groups at 4 months [experimental vs. control: mean 26 (SD 5) vs. mean 24 (SD 6)], which is not shown in Table 8 because of the median (IQR) being more appropriate for skewed data. In addition, there was a significant difference in self-rated recovery favouring the intervention, but this just failed to reach statistical significance in the sensitivity analysis with adjustment for predictors of missingness (IRR 2.18, 95% CI –0.72 to 5.08). There were no other significant differences at 4 months. At 18 months, there was little evidence of any effect, except that the difference in social network size favoured the intervention approached, but did not reach, statistical significance.

Similar numbers of participants in each group reported that they had read the workbook at the 4-month interview [i.e. 133 (84%) out of 158 participants in the control group vs. 142 (88%) out of 162 in the intervention group]. However, more participants in the intervention group reported using it to make written plans. Using the workbook to make written plans was reported by between 28% (38/138 participants for the section ‘Managing your ups and downs’) and 44% (61/138 participants for the section ‘Moving on after a crisis’) of participants in the control group, compared with between 58% (83/144 participants for the section ‘Your goals and dreams’) and 64% (92/144 participants for the section ‘Keeping well’) of the intervention group.

Strengths and limitations

This was a rigorously conducted RCT with blinding of researchers and data analysts, although blinding of participants was not possible. Data for the primary outcome were collected from patient records from blinded NHS staff, although clinicians working with the participants could not be blinded. A power calculation determined sample size. As anticipated when planning the project, an excellent follow-up rate was obtained for the primary outcome when data were collected from patient records, and good follow-up rates of nearly 80% were obtained at 4 months, although they fell below 60% at 18 months.

There is a 64% probability that the intervention is cost-effective compared with treatment as usual, resulting in more QALYs for lower costs, even after the cost of the intervention was taken into account. The difference in costs and QALYs is not statistically significant though, with wide CIs for the costs. This is the main reason that the probability that the intervention is cost-effective is lower than expected, given that the intervention dominates treatment as usual.

One limitation of the study could arise from the way participants were recruited. CRT staff identified potentially eligible service users from their caseload; therefore, it was possible that those recruited were service users who were well engaged with, and easy to contact by, CRT staff. Patient data for five participants at baseline and five participants at follow-up could not be retrieved by NHS IT teams and, thus, had to be treated as missing. A completion rate of > 60% was not achieved for the 18-month follow-up interview because of difficulty in contacting and consenting participants. This was despite efforts to maximise the response rate by offering participants shorter telephone interviews as an alternative to face-to-face interviews. The 18-month follow-up was planned once participant recruitment had been completed, with the agreement and additional funds from the study funder. Another limitation with potential to introduce bias and limit generalisability is that only 24% of those eligible were recruited to the trial.

Recommendations for future research

Research is required to further investigate the critical ingredients and underlying mechanisms of change of this successful intervention. A planned process analysis using data from this study can help with this, exploring the relationship with outcomes of completion of the recovery plan, therapeutic alliance with the PSW and the PSW’s perceived recovery orientation, and the degree of match between PSW and service users on demographic and clinical variables.

Replication studies to corroborate the findings from this trial are desirable. However, the intervention appears successful as a means of reducing relapse in people following discharge from a CRT after home treatment, a result that coheres with the success of other self-management interventions in people with mental health problems.^95,121 The reduction in relapse also translates to a potential reduction in the average cost per patient, compensating the cost of the intervention. There is thus a good case for wider dissemination of such interventions in routine NHS settings, and for research to investigate the best methods of supporting their implementation.

Patient and public involvement within the CORE programme

Service users and carers have been variously involved in the CORE programme:

• a service user consultant/researcher was a study co-applicant
• employed as a PPI co-ordinator
• participated in working groups performing advisory functions
• conducted of a range of programme activities:
  • peer support
  • resource kit development
  • fidelity measure development
  • fidelity reviews
  • qualitative interviews
  • focus group facilitation
  • interview analysis.

The contributors

Interviews were completed with 12 contributors aged between 30 and 76 years. Most were women (n = 9). Although contributors had in common ‘lived experiences’ with mental illness and mental health services, contributors reported varied life histories. The majority reported substantial experience of PPI. Involvement in the CORE programme covered the activities listed in Introduction but varied in duration. Some (n = 7) reported continuous involvement from ‘the beginning’, and a minority had become involved subsequently (n = 2) or withdrew (n = 3).

Contributors shared the view that PPI in research generally aimed to improve processes and outcomes by ‘grounding’ research, enhancing its ethical status and improving feasibility, acceptability, relevance and credibility.

Contributors commonly linked their involvement in the CORE programme’s PPI to personal well-being, the intrinsic rewards associated with making a contribution and an aspiration to improve mental health services.

Patient and public involvement in the CORE programme: the process and practicalities

Patient and public involvement activities

Influence and impact of patient and public involvement on CORE

Most contributors welcomed the opportunity to express views and, although wanting these heard, were aware and accepting of the constraints within which a programme of research, such as CORE, operated.

Some contributors, however, felt constrained, noting that because some working group meetings were convened after the ethics review of the relevant CORE substudy was complete, the scope for amending study procedures was limited by the willingness of the investigators to submit amendments to the Research Ethics Committee (and tolerate any consequent delays).

Although noting that impact was difficult to assess, contributors were typically confident that PPI had shaped or influenced the process of the CORE programme in various ways:

• influencing the CRT best practice model
• making activities more ‘user-friendly’ or respectful (e.g. restructuring focus groups)
• design of the resource kit
• design of the peer-based self-management intervention
• design of study documents (e.g. participant information sheets)
• enhancing the credibility of fidelity reviews
• qualitative data collection.

The experience of involvement

The majority of contributors described their participation as developing interpersonal and practical skills they considered valuable. Many spoke of ‘learning’ about research or the ways health services ‘work’. Contributors commonly spoke of being affected emotionally or psychologically. Sometimes experiences were transient (e.g. ‘being upset’ listening to service users’ accounts), sometimes lasting – contributors spoke of deepening sense of self-esteem and respect, of coming to develop a new (and preferred) sense of self. Some contributors reported that involvement with the CORE programme had facilitated further involvement opportunities.

The majority of contributors described PPI in the CORE programme as robust, meaningful and worthwhile, and generally experienced it as a rewarding and intellectually stimulating activity. Contributors linked the maintenance of their involvement to feeling respected and trusting that their input was valued. However, some (including two who discontinued participation) expressed the view that they and their input were marginalised and reported dissatisfaction with the process and the level of involvement achieved in relation to aspects of research design.

Summary points