Introduction
The preceding three chapters have described SLT-led interventions from the perspective of SLTs, linked with the evidence from systematic research in relation to their effectiveness. The perspectives of children and parents have also been made explicit. The next set of studies, described in this chapter, reports and appraises the tools that are currently used for targeted assessment and measurement of outcomes. The aim was to identify tools that can be used to help target interventions within the typology to meet individual child and family needs and measure the subsequent outcomes.
This chapter is organised into two sections: the first reports findings on the assessment processes used by SLTs to decide whether or not a particular aspect of intervention is appropriate; the second reports on the outcomes that are linked to the typology themes described in Chapter 2 (see Study 2.1: identifying the themes of speech and language therapy practice). Recommendations are made regarding the types of assessment, measurement and evaluation processes that are needed for the future. Before the findings are reported, a brief introduction is provided outlining the issues surrounding assessment and measurement of outcomes.
Assessment
Assessment is the process of collecting reliable and valid information and integrating and interpreting it to form a sound basis for all clinical judgements and decision-making.249 It is stated in Communicating Quality 3,250 the standards and guidance manual for UK SLTs, that ‘All intervention is delivered on the basis of on-going assessment and review of progress with the individual (and/or carer as appropriate) as measured against targeted outcomes’ (p. 35).
This highlights the strong, dynamic inter-relationship that should exist between assessment of need, intervention and subsequent outcomes. SLTs are required to conduct thorough assessments to ensure provision of the most appropriate intervention matched to identified need.165,251,252 In terms of Child Talk, clarification is needed about which assessment processes can help determine which themes from the typology should be included in an intervention.
A number of studies have explored the assessment procedures used by therapists working with children with communication difficulties and what influences the choices they make253,254 and, in particular, what formal (standardised) or informal (non-standardised) assessments are used and how uniform the practice is across the profession.253,255,256 Internationally, a high proportion of therapists report using informal, self-designed assessments.253,254,257,258 Each method of assessment has its own strengths and limitations and, ideally, the determination of the most appropriate intervention should be made on the basis of information collected using a number of processes.250 Formal assessment measures are usually norm-referenced and standardised, enabling children’s scores to be compared with data from typically developing (TD), same-aged peers. However, published assessments may lack a professional evidence base and independent verification of concurrent and construct validities because of the high costs and time involved.259 On the other hand, informal measures have not undergone a standardisation process. These measures tend to be developed for a specific population or investigative purpose and do not have their own normative data. Findings from these so-called ‘informal’ assessments are interpreted by therapists with reference to therapists’ knowledge about functional or developmental norms or with reference to other relevant theories about speech and language development and impairment gained through training, from the literature and from experience.260 Limbrick et al.253 acknowledge that, although many informal therapist-generated measures lack scientific rigour, positive features include being user-friendly and appropriate for the intended population and context.
Although there is an acknowledgement in professional literature that different methods are needed to provide a comprehensive picture of a child and his or her family, there is wide variation in the exact combinations of assessment processes that are used in any particular instance. Studies have found that, although there are similarities in the parameters assessed, therapists differ in their choice of procedure,63,254 and this seems to be related to clinicians’ beliefs and theoretical perspectives.261 In line with UK findings, the choice appeared to be influenced primarily by financial constraints although factors such as validity and reliability, versatility, efficiency, ease of administration and familiarity are also important.132
Outcomes
Context of speech and language service provision in England
There has been a significant change over the past 10 years in the information required from providers of education, health and social care, not only about what services are offered but also about how well they are delivered and with what outcomes. This information is necessary not only for quality assurance and the evaluation of services, but also, most importantly for service users and the public, so that all can make informed choices about their care.262–264 In the UK, health-care provision is now based on a model of commissioning services that is measured against the NHS Outcomes Framework.264–267 The quality of provision is judged against three broad indicators: (1) safety, (2) importantly for this research programme, the effectiveness of the treatment and care provided and (3) patient (service user) experience, measured by both clinical and patient-reported outcomes. Thus, in developing a framework of interventions for preschool children with PSLI, it is crucial to show the links between the interventions and the outcomes. Either on its own is insufficient,268 although when interventions are complex, involving large teams and multiple contexts, identifying the outcome of any particular contribution to an intervention is challenging.269
In the context of health care, outcomes should reflect change resulting from therapeutic interventions and should represent a key aspect of assessing the effectiveness of care.270 They have been defined as ‘a predicted measure of change that demonstrates a valid and significant therapeutic impact following an agreed intervention’271 (p. 3).
Outcome measurement seeks to capture the benefits of interventions in ‘real-world’ conditions, particularly functional (positive) change for the individual client.272 However, the development of ‘functional outcomes’ for speech and language therapy that are consistent, psychometrically sound and clinically meaningful has been both problematic and long awaited.255,273,274 Over the last 20 years there has been significant investment in terms of time and finance to develop robust outcome measurement systems in speech and language therapy, notably the Therapy Outcome Measures (TOMs)275–277 and the American Speech–Language–Hearing Association (ASHA) National Outcomes Measurement System (NOMS).255 Although both are widely used, there is by no means universal take-up of either system. Indeed, recent surveys suggest that it is not even clear to many practitioners what constitutes an outcome measure.89 Roulstone et al.89 found that only one-third or 33% of therapists working in England were delivering any outcome data to their manager and, of those, under half referred to systems that used criterion-based measures such as checklists or standardised (norm-referenced) tests.
Various sources provide recommendations regarding the domains that can be related to interventions for speech and language impairments. For example, in Communicating Quality 3250 it is suggested that the outcomes from speech and language therapy that contribute to health, educational and psychosocial benefits will include:
diagnosis of communication and/or swallowing disorders
maintenance of optimal communication and/or swallowing abilities
improvement in the speech, language and communication abilities of individuals
improved use of existing function
reduction of communication anxiety and avoidance
improvement in interaction and effective social communication
increased awareness of others about communication and/or improved communication environment
greater opportunities for communication.
The Commissioning Support Programme (CSP)273 suggests three types of outcomes from speech and language service provision that commissioners should consider:
the users’ reported experiences of services that they have received (patient-reported outcome measures)
the achievement of therapy or intervention goals
the directly measured impact of services on users’ speech, language and communication skills, attainment and well-being.
From the BCRP223 a number of outcome themes were identified as important by, and for, children:
time for fun and laughter, both in terms of social activities and in relationships with teachers and family
feeling supported and listened to
dealing with emotions, particularly feelings of frustration, anger and sadness
improving other people’s behaviour towards them in terms of listening more and interrupting less, teasing and shouting.
Parents in the study identified, and understood, the vital role of successful communication in supporting their children to achieve the ‘fundamental outcomes’ of physical and emotional well-being and staying safe, but the two key outcomes were for their children to achieve independence and social inclusion.
Although there are a number of tools available for the measurement of outcomes, there are also considered to be gaps in relation to certain groups. For example, a lack of available resources to demonstrate change and development of communication skills for individuals who use augmentative and alternative communication (AAC) prompted a review of outcome measurement tools by a multidisciplinary group of members of Communication Matters.278 They reviewed 23 measures applicable across the lifespan including those addressing quality of life, functional abilities, goal attainment, functional health and well-being, self-image and self-esteem, self-perception, occupational performance, environmental technologies and communicative competencies. They provide readers with a comprehensive overview and thorough description of each measure including the focus, administrative procedures and psychometric qualities and whether or not it is an assessment or a measurement tool, if it is based on a specific framework, whether or not data can be aggregated and the appropriateness for AAC. Following the study of parents’ and children’s perspectives on outcomes, Roulstone et al.223 reviewed 27 self-report instruments to identify those that measured the outcomes of interest to parents and children. A critical appraisal of those instruments identified nine measures of quality of life, functional communication and socioemotional functioning that were considered to be relevant and robust.
Therefore, there are a number of potential measures available; however, the direct link between particular interventions and their related outcomes has not been identified.
The purpose of this study is to explore and achieve consensus on what are the best ‘outcomes’ and ‘outcome measures’ for interventions for preschool children with PSLI. This phase of the research programme has focused on outcomes specifically related to interventions in the typology framework.
Discussion
The need to demonstrate what works in terms of the outcomes of speech and language therapy is not a new phenomenon.274 Following the Bercow review,28 the CSP produced a set of documents designed to support joint commissioning and an appreciation of the perspectives of children and young people and their parents. Three documents are now available on the website of the Communication Trust [see www.thecommunicationtrust.org.uk/commissioners/slcn-commissioning-tools (accessed 30 March 2015)]. Despite this, there is still a lack of consistency in the way that services are commissioned and provided and in how the subsequent outcomes are measured. Surveys have also shown that the SLT profession is not currently collecting, or required to report, any consistent information nationally relating to the outcomes of intervention.287 This is compounded further by the lack of inclusion in the NHS Outcomes Framework263 of any outcomes pertinent to children with SLCN and of the services that they require. This is particularly concerning given that one of the three main purposes of the NHS Outcomes Framework is ‘to provide an accountability mechanism between the Secretary of State for Health and the NHS Commissioning Board [now NHS England 2014] for the effective spend of some £95bn of public money’ (p. 4).263
Without recognition, or appropriate evidence, of the contribution that speech and language therapy makes to the lives, and life chances, of children and families, resource allocation is likely to diminish. This is particularly relevant as the commissioning of all public services, including education and health, is now in an era of ‘value-based’ care.288 In addition, given the current and future economic position of public services funding, the main driver for all commissioners is ‘maximising value for patients: that is, achieving the best outcomes at the lowest cost’ (p. 51).288
Porter and Lee288 acknowledge the challenge of the change involved in moving to outcomes-driven provision, given decades of entrenched practices. They urge for changes to come from within the professions because, ultimately, value will be determined by how therapy is practised.
This chapter has explored what is currently known about the current, and potential, measurement tools and outcomes linked to the typology themes for children with PSLI. The research first considered what combinations of measurement tools best identify appropriate interventions, in relation to the typology themes. Published, norm-referenced assessments are considered essential diagnostic tools and should have standardised administrative, scoring and interpretative procedures with established validity, reliability and applicability for specified populations, in this case children with PSLI.259,289 However, diagnosis is not the only purpose of assessment. Therapists need to draw on a range of information sources to support their clinical decision-making, including what to prioritise when a number of aspects require intervention (specifically in this research, which typology themes to target), how to track progress, how to determine when intervention is no longer required and how to measure outcomes. Some of the influencing factors are ‘child based’, such as developmental level and profile, ability to engage, learning style and preferences and level of severity of the presenting difficulty. These features would be the focus of assessment for the majority of the typology themes (including ‘speech’, ‘expressive language’, ‘comprehension’, ‘functional communication’, ‘foundation skills’, ‘self-monitoring’ and ‘generalisation’). Other factors can be considered as ‘environmental’, including the level of understanding of significant adults and their ability to support, adapt and ‘scaffold’ the child’s learning and provide opportunities for communication. These would be investigated mainly with respect to the typology themes of ‘adult understanding’ and ‘adult–child interaction’. It is acknowledged that, optimally, therapists should incorporate data from both quantitative and qualitative sources to fully examine the abilities of any child97 and provide a holistic picture. WHO283 suggests that comparing ‘capacity’ and ‘performance’ can provide useful information about how the environment can be adapted to improve an individual’s performance in functional activities. A dynamic assessment approach involves evaluating a child’s ability to perform activities when provided with additional cues or ‘scaffolding’, that is, when the linguistic context is modified. This dynamic assessment process can reveal a child’s ‘learning potential’ and capacity for change at an individual or an environmental level but may be difficult to carry out in a reliable and valid way.250 It also provides valuable information about which strategies will best help a child to achieve and facilitate the transfer, or generalisation, of skills to other environments.290 This approach is particularly valuable for SLTs who focus much of their intervention on enhancing the child’s communication environment by supporting parents and significant others to help the child progress. However, therapists report a limited number of assessments that provide evidence of children’s communication performance in everyday environments.132,253,291–293
In terms of which assessments or methodologies were found to be routinely employed to identify therapy aims, practice is not consistent across services or therapists. We found that therapists use mixed methods of data collection to provide information about children’s communication in both ideal (capacity) and functional (performance) contexts.283 This included nine naturalistic or informal methods of data collection that were used to determine how children performed functionally day-to-day and to determine the level of knowledge and understanding of significant adults in their communication environments, either at home or in a care setting. These methods yielded information relating mainly to six of the typology themes: ‘foundation skills’, ‘adult understanding’, ‘adult–child interaction’, ‘self-monitoring’, ‘generalisation’ and ‘functional use of language’. Standardised and norm-referenced assessments were used to gather information particularly about a child’s developmental level and the characteristics, or profile, of the presenting communication difficulty or impairment, such as its severity. The use of standardised assessments tended to focus mainly on three typology themes – ‘comprehension’, ‘speech’ and ‘expression’ – although these assessments were also used to a lesser degree in relation to ‘foundation skills’ and ‘functional use of language’. Without exception, for all typology themes, naturalistic or informal data were used to supplement formal assessment findings to determine children’s needs and to measure the outcomes of intervention, which is in line with earlier findings.253 However, of the standardised measures in common use, only the CELF-4 met all of the psychometric quality standards as specified by Friberg.97 In addition, therapists reported administering only subsections of assessments that they felt were relevant to a particular area and they rarely completed the whole of an assessment. This affects the sensitivity (accurately identifying the presence of impairment) and specificity (accurately identifying the absence of impairment) of the assessment findings and potentially leads to over- or underdiagnosis. As a result, some children may not receive early or appropriate intervention that could optimise their outcomes.294 Alternatively, some children may receive intervention that may not necessarily be required and therefore resources may not be being targeted as effectively as possible. It also raises questions about validity and how the findings are used to determine need, eligibility, intervention, evaluation of outcomes and comparison across services.97,253,258,259 Therapists using assessments in this way are not necessarily using the standardised scores; rather, they seem to be using the assessments as a way of standardising the sampling context and as such the assessments supplement other informal assessment processes. This too has the potential to undermine the reliability of the results across children.
The research programme also considered the measurement of outcomes for preschool children with PSLI. To capture outcomes that are relevant to all stakeholders, including children, families, therapists, commissioners and quality assurance agencies, outcome measures need to be multidimensional and multilevel.274,295,296 This will include qualitative and quantitative data that are valid, reliable, sensitive to change, specific, comparable and understandable and which have a particular focus on functional changes for the individual child and family.274 One tool would be unable to capture data for all purposes; hence, the NHS Outcomes Framework outcome domains are underpinned by a range of indicators, which, when the data are aggregated, will provide evidence of how well the overarching outcome has been met at the individual, family, service, community, society and national levels.263
A large number of standardised measurement tools exist that can provide information pre and post intervention about children’s communication (capacity) in ideal conditions, particularly at the ‘impairment’ level. However, in this study we found that therapists consistently reported that intervention across the typology themes can impact on much wider aspects of children’s lives. Fifteen outcome domains emerged from the nine themes. Some are common across a number of typology themes, such as emotional well-being (all), participation and inclusion, social interaction, behaviour and educational achievement. However, others are specific to an intervention or to a typology theme; these are adult–SLT partnership, adult knowledge and use of strategies, adult–child relationship, attention and listening, communicative competence, intelligibility, self-awareness, and understanding and use of language. We therefore have a picture of the PSLI intervention typology contributing to a range of outcomes impacting on the child and his or her family, eight of which are typology theme specific.
Findings from the BCRP223 identified two overarching outcomes that parents wanted for their children: to be able to achieve independence and social inclusion. Therapists in this study believe that their interventions do impact on these two outcome domains. Children in the BCRP were concerned about emotional well-being and social interactions; these outcomes were also identified in this study, demonstrating a high level of consensus between service users and therapists on desired outcomes. However, as already highlighted, measurement and data gathering practices are inconsistent and currently focus predominantly on the presenting area of difficulty, for example sound production, and/or hypothetically contributory factors, such as styles of ‘adult–child interaction’. As it is now necessary to be able to demonstrate the wider impact of SLT-led intervention, the desired outcomes must have a starting point at the beginning of intervention to demonstrate change over time. This requires measurement tools and specific outcome indicators that are reliable and valid to provide appropriate supporting evidence.
A review of existing tools was undertaken to identify any that might be relevant to the measurement of outcomes related to the typology themes. As already indicated, a number of formal measurement tools were identified in the study that are appropriate and which are frequently used to identify the need for intervention at the impairment level for individual typology themes, including ‘comprehension’, ‘speech’, ‘foundation skills’ and ‘expression’. However, in line with international consensus, there were felt to be few reliable measures to assess children’s performance, that is, functional use of speech and language in their everyday context rather than ‘capacity’ under controlled or ‘ideal’ test conditions.297 Of particular note was the lack of common valid and reliable tools to specifically identify the need for intervention in PSLI for the themes of ‘adult understanding’, ‘adult–child interaction’ and, to a lesser degree, ‘generalisation’ and ‘functional use of language’.
Tools appear to be particularly lacking for measuring specific outcomes related to outcome domains. So, for example, therapists reported that interventions would impact on outcome domains but no mechanism or tool was available to measure the outcomes. This included outcome domains such as adult–child relationship, adult knowledge and use of strategies, adult–therapist relationship, behaviour, emotional well-being, communicative competence, independence, intelligibility, participation and inclusion, self-awareness and social interaction. In terms of measuring outcomes in relation to the adult–child relationship and adult knowledge and use of strategies, a new common tool is needed that is of relevance to PSLI. In some areas, tools are available. For example, there are a number of useful tools in existence for intelligibility including the Intelligibility in Context Scale258 and the Children’s Speech Intelligibility Measure (CSIM).298 The adoption of a common measure requires further consensus work. Many of the other domains are encompassed in the WHO International Classification of Functioning, Disability and Health (ICF) – children and youth version, which is known to be both valid and reliable.283 There are two tools based on this that have been developed to capture information specifically in relation to communication. They are the TOMs275 and the Focus on the Outcomes of Communication Under Six (FOCUS) measure,299 either of which would be appropriate. Again, further consensus work is required with the profession. In relation to the adaptation of existing tools, this would require a considerable amount of resources to achieve agreed content, reliability and validity.
Further consideration was given to the principles that should govern further tool selection and use. A combination of core quantitative and qualitative valid and reliable measures is considered to be a priority for service development and delivery to capture consistent data that are specific, allow accurate diagnosis of the presenting or potential difficulty, highlight the relevant typology theme(s) and provide a baseline from which to measure progress and achievement of desired outcomes. A core combination of measurement tools should ideally identify:
relevant information from parents and significant others (adults) who know the child well
relevant information about the child and his or her environment – including the level of support for the planned intervention
information about the impact of the presenting PSLI on the child’s current day-to-day functioning and participation and academic achievement
diagnostic information about the severity of the presenting difficulty and the potential for change
which typology theme requires intervention
when progress is adequate and further intervention is not necessary
appropriate measurable outcomes.
This would provide the basis of a core national data set to allow monitoring of progress, comparison of intervention approaches and provision of information to service users, commissioners and researchers.
In summary, the study has identified a number of measurement tools and methodologies that therapists might use to focus intervention on the nine typology themes and measure outcomes, mainly at the impairment level. Fifteen outcome domains have been identified that link to the typology themes, eight of which are theme specific. There is a need for a core set of current and, additionally, valid and reliable measures to focus intervention appropriately on the typology themes. These should also provide evidence-based outcome indicators, particularly in relation to the functional ‘real-life’ impact of interventions.
