Valued outcomes

There exist recommended procedures for agreeing what should be a core set of outcomes in various fields of health care. As Williamson et al.¹⁸ note, ‘insufficient attention has been paid to the outcomes measured in clinical trials’. Consistency and interpretation will be improved if researchers always collect and report on core outcomes. The Core Outcome Measures in Effectiveness Trials (COMET) initiative funded by the Medical Research Council Network of Hubs for Trials Methodology Research aims to develop a set of evidence-based procedures for developing a core outcome set. The suggested steps involve:

• Step 1 Agree the scope of the area of health care.
• Step 2 Identify existing knowledge about outcomes.
• Step 3 Involve key stakeholders, including patients and health-care providers.
• Step 4 Develop consensus about what to measure. Techniques for doing this in an inclusive and objective way are outlined in Williamson et al.,¹⁸ including how to determine when consensus has been achieved.

A systematic review of studies that aimed to determine which outcomes to measure in clinical trials in children concluded that in most specialties no research had been undertaken.¹⁹

The scope for this review was determined in the Health Technology Assessment (HTA) commissioning brief (i.e. COMET step 1) and includes a potentially broader use of outcomes than only in trials. This chapter presents the work undertaken on steps 2 and 3, i.e. to identify priorities for child outcomes as valued by parents and professionals, and as explored in qualitative literature. Because of the complexity of ASD as a disorder, and the developmental context of measuring outcomes up to the age of 6 years, the MeASURe review adopted a further step of placing the findings of the consultation stages in a conceptual framework to guide the full review of tools for measurement. This framework also guided further consultation with stakeholders about the relative importance of outcomes to measure. MeASURe did not undertake a further formal process to develop consensus (step 4 above). It may be that ‘consensus’ would be hard to achieve but it would require further procedures. In principle, the choice of outcomes to focus upon depends on the specific research question being asked, and on what is important to particular groups of stakeholders.

Considerations for developing a conceptual framework in autism spectrum disorder

One important potential basis for a conceptual framework for valued outcomes for children with ASD is the International Classification of Functioning, Disability and Health for Children and Youth (ICF-CY),²⁰ so that what is measured can be ‘mapped’ against domains of functioning (e.g. Learning, Communication, Self-Care) and participation (e.g. Relationships, Community Life).

The conceptual framework should also be influenced by an understanding of ASD. The behavioural characteristics of ASD are underpinned by genetic, brain structure and neuropsychological differences from typical development.²¹ The conclusions of many studies have led to the revision of the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM-5) published in May 2013, such that the impairments in ASD are best considered within two groupings: social/communication difficulties and restricted/repetitive interests and behaviours, including hypo- or hyper-responsiveness to sensory stimuli. The aetiological underpinnings for each of these broad domains of impairments may be different, and both may be targets for interventions targeting ‘core’ features of autism.

Another aspect of complexity in the understanding of ASD is that its measurement is affected by developmental considerations, i.e. children’s profile of skills and difficulties may look very different over time, and those trajectories will also be affected by levels of ability. As an obvious example, we cannot measure children’s social ability to make and keep friendships with validity until they are of an age at which that might be expected in typical development. However, there is some recent agreement on the core early impairments that may be observed. By the age of 2 years, differences in the development of children with ASD (from typically developing children and those with developmental delay) are evident in behaviours such as fearfulness, frustration and lack of co-operation, quick mood changes, and fine and gross motor skills.²² This knowledge has been enhanced by recent studies of the early development of baby siblings of children with autism (who have an increased chance of themselves developing ASD); for example, Zwaigenbaum et al.^23,24 reported unusual eye contact, a lack of visual attention, orientating to name, imitation, social interest and emotional affect, and heightened sensory-orientated behaviours. These combinations of deficits have consequences for development of relationships, early language and play, and, in turn, for the content and targets of early intervention.

The complexity of understanding ASD is made even greater when considering the interaction between domains of development, and how deficits in one may impact upon another; for example, visual sensory overload may lead to avoidance, which reduces opportunities for visual learning and social experience, leading to poor social skills. Furthermore, there is evidence that pragmatic skills (such as social timing in interaction) are closely associated with particular types of behavioural profiles.^25,26 Thus the conceptual framework for a review of outcome tools should consider both measurement of individual areas of functioning, which are likely to change over time, and also tools that bring together these separate areas into a more holistic assessment. It is particularly important to be able to create profiles for children with ASD, who often have difficulties in generalising learning between areas of skill and also generalising skills from one setting to another.

It is also important to detail other associated difficulties that are not unique to ASD but which, nevertheless, can play a major part in children’s development and the burden of care for families. These commonly include feeding and eating difficulties (resistance to certain food textures, faddiness about types and colours of food, etc.), behaviour and sleeping problems. Children who lack adequate nutrition and sleep are likely to be bad tempered and even more rigid in their thinking and behaviour. Furthermore, adaptive functioning may be more impaired in children with ASD than would be expected from their level of ability. Finally, as young children’s development is intimately affected by their environment, including the health, skills and resilience of parents and carers, it is important to include consideration of the impact on the family.

Although the conceptual framework developed over the process of consultation, and was informed by the separate activities described below, it is presented first for brevity and clarity (Table 1). The framework adopted was informed by discussion within the project team, and inspection of other relevant frameworks such as groupings of target symptoms/skills from interventions studies in ASD,^20,27 education outcomes,²⁸ grouping of interventions by Research Autism (http://researchautism.net/pages/autism.treatments.therapies.interventions/) and compilation of measures for children with developmental disabilities.²⁹ One area discussed was how to categorise quality of life, which is essentially a construct separate from the ICF-CY. The decision was made to include it in the participation domain, as it implies how an individual interacts with their environment.³⁰ For the MeASURe conceptual framework, there are four primary domains, with subdomains in each.

TABLE 1

Conceptual framework for the MeASURe review

Question

What child- and/or family-specific outcomes do parents of children with ASD perceive as important?

Methods

Data synthesis

In order to present an overview to the parent advisory groups and the research team, key findings (including illustrative quotes) and analytical frameworks from each paper were extracted and tabulated, and themes identified.

Results

Searches identified 102 relevant papers. Fourteen studies were selected as sufficiently relevant to obtain a full text of the paper. Three of these were excluded because they did not collect qualitative data on outcomes; four were excluded because they contained no data on outcomes. Seven articles remained.

It was clear from an initial inspection of these articles that the quality of data was variable and ranged in focus/topic. Three articles reported data relevant to only parent outcomes,^31–33 including process outcomes. Three articles reported data relevant to child outcomes only^34–36 and one reported both.³⁷ One study included data collected directly from children and young people with ASD,³⁴ although original quotes from children were not presented.

The age range of children represented in these studies was 0–21 years. Just two studies^31,33 focused specifically on younger children (3–6 years;³¹ up to 5 years³³). The diagnoses represented in the studies typically relied on parental reports. Two studies^35,37 focused on particular diagnostic groups (Asperger syndrome;³⁷ ASD with no functional communication³⁵) and others were defined in terms of use of a particular service (speech-and-language therapy;³¹ assessment and diagnosis;³² preschool educational intervention³³).

Data collection methods included focus groups, individual face-to-face interviews and open-ended/free-text questions within postal and web-based surveys.

Although we did not appraise quality of studies formally (using any standard checklist), the quality of reporting sampling and recruitment, data collection methods and data analysis processes was extremely variable.

Given the significant limitations, in terms of quality and relevance, a ‘light touch’ data extraction was undertaken to identify outcomes and themes (see Appendix 2).

In terms of child outcomes, it was notable that some aspects deemed ‘fundamental’ by parents may not be regularly assessed (such as ‘safety’),³⁴ and certainly not as an outcome of an early intervention trial. ‘Awareness of danger’ was added to the conceptual framework (subdomain Social inclusion) as a fundamental issue of safety. The parents’ and young people’s emphasis on participation outcomes (such as being ‘isolated from peers’ or ‘live a normal life’) may also not be reflected in what is usually measured. Constructs concerning child and parent stress, and positive mental health^36,37 were supported as important to include as outcome constructs in the conceptual framework.

In addition, parents often highlighted the processes of interaction with professionals, and the utility of information from assessments. Parents expected the service to provide them with information and research literature; to involve them in decision-making processes; and to teach parents how to deliver therapies at home.³¹ Braiden et al.³² reported that parents ‘desired information relevant and applicable to their child to assist them in understanding and making sense of their own child’s presentation’. They also mentioned wanting to have positive times with their child: ‘when he is behaving well and not gearing up for a fight, he’s a very happy and pleasant child’.³⁷ Such parent priorities have informed the conclusions of the MeASURe project.

Questions

Methods

People on the autism spectrum in Birmingham and Newcastle were approached for their opinions by a person they knew well. In Birmingham, six adults were known to the University and responded by e-mail; 10 children were approached by a member of the Autism Outreach Team and were selected on the basis that they were thought able to give their views on the questions. In Newcastle, two children attending a National Autistic Society social group responded in person and, likewise, two young adults attending a social inclusion group. Responses to the questions were thus received from 12 young people aged 9–15 years, and from eight adults aged 22–43 years. Each respondent was given a shopping voucher in acknowledgement of their contribution. The verbatim responses were collated and common themes extracted.

Results

Question

What constructs are most often measured by early years professionals in monitoring children’s progress?

Methods

A survey was undertaken in autumn 2012 through the British Academy of Childhood Disability database of 240 UK Child Development Teams. Professionals were sent an electronic link to a web-based survey that took 10 minutes to complete. In parallel, education professionals received the survey via (1) ‘4 Children’, a national charity and UK Government strategic partner for early years and child care, through their database of 15,000 Early Years providers across England; (2) the database of an independent specialist centre for early years children with ASD; and (3) 150 practitioners undertaking the Birmingham University School of Education Autism Programme residential weekend.

The survey had five sections. Respondents were asked:

1. About their profession, and the setting in which they work.
2. ‘Do you regularly work with children on the autism spectrum (this includes any activity that aims to improve/change an area of functioning)?’ and ‘Do you ever measure the progress or outcomes of children on the autism spectrum (i.e. more than just seeing the child once for assessment)?’. Those who indicated ‘yes’ were given access to the rest of the questions.
3. To identify all areas in which they formally measure or informally make judgements about progress or outcomes for children on the autism spectrum whom they see more than once, and who are aged ≤ 6 years. The 68 outcome constructs included were taken from the conceptual framework.
4. To indicate how frequently (on a six-point scale) they used different types of tool: standardised measures of progress or outcome (with manual and comparative age-related information); non-standardised measures (either published or created locally); informal judgements.
5. To give the three areas in which they most frequently measure progress over time, or outcome, with which of the three types of tools.

Results

The 836 respondents included 167 health professionals (paediatricians, speech-and-language therapists, clinical psychologists, occupational therapists, physiotherapists, dietitians, health visitors, social workers and educational psychologists), 353 education professionals (teachers, special educational needs co-ordinators, autism education advisors, teaching assistants, intervention practitioners), 125 nursery nurses and 191 other professionals, many of whom were childminders. Professionals worked in a variety of settings. Many health professionals worked in child development centres or hospitals but some were based mainly in educational settings. Education staff were from mainstream and specialist schools or early years settings.

Five hundred and thirty-seven professionals monitored the progress or outcome of children who were seen more than once, and were able to access the remainder of the survey.

Professionals were more likely to measure characteristics such as amount of speech (76%), social interaction (90%) and attention (79%) than life or adaptive skills (measuring for, or trying on, clothes 6%, difficulties with appointments, e.g. hairdresser, dentist 16%, use of knife and fork 29%), features related to ‘quality of life’ for the child (quality of life 21%, happiness 42%) or the family (nature of sibling relationship 18%, family quality of life 22%, impact on the family 33%).

Professionals were more likely to use their ‘own informal judgement in discussion with parents or other professionals’ than standardised measures to rate improvements (442 respondents agreed with ‘used often’, ‘most of the time’ or ‘always’ compared with 253 who checked ‘never’, ‘rarely’ or ‘sometimes’). The specific types of measures used varied very widely due to the broad range of professional respondents. However, consistently across the questions, around one-third of respondents replied that they used standardised measures, and half said that they were most likely to use parent or professional impression to gauge progress or outcome. (The standardised tools identified as used most frequently were later included in searches in Chapter 4.)

In conclusion, this survey found that professionals are most likely to measure features related to core impairment subdomains of autism, rather than aspects of daily living, family functioning, and child well-being and happiness.

Question

What outcomes do parents consider as important to be assessed?

Methods

Parent advisory groups were recruited at three sites (Exeter, South London, Newcastle). In Exeter, the Peninsula Cerebra Research Unit involves families of disabled children as partners in research through a Family Faculty.³⁸ Parents of children with ASD were e-mailed and invited to volunteer: 12 expressed interest and seven participated in one or more meetings. In London, the Newcomen Neurodisability Team involves families of children with ASD in giving advice on an ad hoc basis; for MeASURe, 10 parents were invited by e-mail and six participated in one or more meetings. In Newcastle, parents of children with ASD aged ≤ 10 years were invited by e-mail; four participated in one or more meetings. Thus a total of 17 parents were involved in discussion meetings. Parents were given a financial acknowledgement in addition to travel expenses, to recognise their time and expertise at each attendance. Meetings were held at three points during the MeASURe project.

Early meeting To explore parents’ priorities and experiences of assessment and identify what outcomes parents saw as important, especially for monitoring their young child with ASD over time. This session involved an explanation of the aims of the project and open discussion, led by a member of the project team and a parent involvement co-ordinator.

Mid-point meeting To undertake a Q-sort of constructs emerging from the conceptual framework. Two members of the MeASURe project team (NL, GM) created ‘lay wording’ versions of the constructs. Sixty-two constructs were presented on cards in a jumbled order (i.e. not including symptom severity, physical indicators, global measure of function, global measure of outcome). The way in which the constructs had been chosen was introduced by the project team member. Through discussion within the parent group, the constructs were sorted on to a ‘forced-choice’ grid in a pyramid shape on a large piece of paper. Columns on the grid were rated for levels of importance (from ‘more’ to ‘less’ on an 11-point scale), i.e. ‘the importance of various things which could be measured when tracking the progress of children with autism aged up to 6 years, or in measuring the outcome of a specific preschool intervention’. It was stressed that none of the constructs was considered unimportant.

End-point meeting Parent groups met again to consider a summary of the findings of the literature reviews and early consultations. This included a question about the reasons for differences between what parents consider important to be measured and what professionals most often measure. The main activity was to examine five questionnaires that had been rated positively in the systematic reviews. Parents were asked to compare and contrast two questionnaires about parent stress, two questionnaires about children’s behaviour problems, and one questionnaire designed as a global measure of outcome. The issues raised were then taken by parent representatives to the MeASURe project Discussion Day on 14 February 2014 (see Chapter 2, Consultation with multiple stakeholders).

Early meeting

Parents expected that professionals would focus on assessment of core features of autism, such as communication and social interaction. However, they suggested that the child’s skills should be acknowledged and more attention be paid to unusual behaviours that the child is exhibiting, as well as measuring what the child is not achieving. For parents, priority areas for measurement included habit behaviours (such as sleep, diet and food-related behaviours, sensory processing issues, toileting) and also challenging behaviours and ‘meltdowns’ (such as self-harm, hitting out, anxiety, stress, happiness, tics). Parents endorsed the importance of social communication and social functioning (interacting, playing with others, playing alone, understanding and communicating) and, furthermore, the building blocks of learning, independence and life skills (reading and academic achievements, hobbies and sport, imagination and creativity, self-care, preparing food, getting dressed, time management, vulnerability and danger). They also stated that they recognised that some activities/skills may not seem that important or be seen as relevant for this young age range but become a more significant priority later on in development and as their child progresses through school. Parents also mentioned difficulties they had with taking children to appointments for health care (vaccination, dental care, shoes, eyes and hearing). These constructs influenced the conceptual framework, and the content of the survey for professionals.

Parents also commented on aspects of the process of assessment. They recommended the use of video in relaxed environments, so that professionals may watch for changes. They stressed the value of information gathered in a range of settings (suggesting use of video to rate change over time and between different settings). Figure 1 illustrates the themes arising from one of the parent group meetings.

FIGURE 1

Themes from Exeter parent group discussion. DLA, Disability Living Allowance.

Mid-point meetings to undertake the Q-sort

There were four groups that completed this task (two in London to accommodate parents’ availability). Taking an average of the Q-sort ratings from all sites, the items rated on average as ‘more important’ can be grouped as:

• Body functions/impairments Hypersensitivity, anxiety, unusual fears, distress, non-verbal ability, expressive and receptive language.
• Activity level indicators Aggression, sleep problems, school readiness.
• Participation Happiness, self-esteem, relationships with brothers and sisters, being bullied/rejected, no awareness of danger.
• Families Parent stress.

The highest level of consistency in rating these constructs across groups was for aggression and sleep problems. Parents rated happiness as important for all young children but one group did not agree that this could be considered an ASD-specific measurable outcome. In discussion parents mentioned that they had had to learn about autism, and so had not understood at the start of assessments of their child why skills such as ‘joint attention’ were of importance.

End-point meetings

In London it was not possible to arrange an end-point meeting; there were a number of barriers for parents’ attendance including ‘travel time’, ‘difficulties getting child care’ and ‘need to battle the new school statementing system’. In Exeter, a preliminary meeting was held to discuss with parents how to assess the strengths and weaknesses of tools used in assessment (i.e. explanations of terms such as reliability, validity and sensitivity to change).

Within the preliminary report, the ratings on importance by parents were compared with the constructs most often measured by professionals (Table 2), and parents were asked to reflect on the differences.

TABLE 2

Difference between professionals and parents on their top 10 constructs for measurement of progress or outcome

The only area of overlap between parents and professionals was ‘challenging behaviour’/’fighting, hitting others’. Parents in Newcastle highlighted that they believed that parents tend to focus on broader outcomes than professionals, as they see their children everyday in different environments. Anxieties and distress were emphasised; parents remarked that it is emotional needs that impact on the child’s and family’s quality of life. Parents also believe that professionals tend to be unaware of these important difficulties before a child enters the social environment of education.

The Exeter parents had a rather different way of viewing the table. They suggested that professionals measured aspects that were intermediate outcomes, which facilitate what parents rate as important. For example, they suggested that parents’ identification of ‘fighting, hitting others’, ‘distress’, ‘happiness’ and even ‘parent stress’ could be mapped from what the professionals highlighted as ‘challenging behaviour’. Similarly, when parents highlighted ‘relationships with brothers and sisters’, these benefited from adequate ‘speech’ and ‘understanding of language’. So, despite the different labels, there was general support from parents for what professionals measure, and parents had noticed their children making progress in these areas.

Question

What outcomes is it important to assess when monitoring the progress and outcomes for children with ASD up to the age of 6 years?

Methods

A Discussion Day was held in London on 14 February 2014. Twenty-five participants were invited: four parents of children with autism; three young people with autism, two of those with staff who support them in education; eight speech-and-language therapists, occupational therapists, paediatricians or psychologists; and two researchers working with children with autism; six MeASURe project researchers who work in health or education services also attended.

As one activity, groups of similar background carried out a further Q-sort to rate the importance of constructs, and ascertain similarities or differences between what parents, professionals and researchers consider most important. The set of 21 constructs for the parents and for the young people was drawn primarily from those reported as most often assessed by the early years professionals in the survey. The set for the professionals and the researchers included the 10 rated as most important by the parent groups, and the 10 most often measured by professionals as reported in the survey. Both sets were completed with added constructs to represent a wide span of subdomains.

We hypothesised that:

• The young people and parents might well agree on the top 10 constructs.
• The professionals might also agree with parents, even although what they actually measure is not in accordance.
• The researchers might choose a different set (more based on intervention elements, symptoms and everyday function).
• We had no expectation about parents’ views on the order of importance of what professionals tend to measure.

Results

Adverse weather conditions and train cancellations prevented several participants joining the Discussion Day, including two young people on the autism spectrum. However, four groups of four people each considered the constructs.

One young adult on the autism spectrum joined the parent group; his/her ranking showed a high level of agreement with the averaged ranking of constructs undertaken previously by parent groups (Spearman rank correlation r_s = 0.618). Fine motor skills were rated higher than previously because of the experiences of the young adult as a child. ‘Friendships’ was rated higher than previously, reflecting on the precursor skills needed by the child early on that will lead later to being able to make friendships. Aspects that affect the emotional state of the child, including sensory processing, continued to be rated highly. ‘Participates in mainstream activities’ was rated low: the group thought ‘this means the ASD child has to adapt to the mainstream world rather than ‘mainstream’ adapting/understanding/respecting ASD needs’. They also gave a low rating to ‘not cooperating, throwing, spitting, won’t sit (challenging behaviour)’, as they considered it the role of adults (parents, education and care staff) to try to make the environment right for the child so his/her autism was less ‘disabling’.

The two multidisciplinary groups of health and education professionals, and the group of ASD researchers, had low agreement with the averaged ratings of the parent groups (r_s = –0.268, 0.131 and –0.063, respectively). The health and education professionals commented that they measure what they can (in the setting, given the available tools) and what they traditionally have done. They emphasised as ‘important’ what they see as most urgent to try to change, such as challenging behaviour and communication skills. In contrast, although acknowledging the importance of the construct ‘positive views of self (self-esteem)’, they gave it a lower rating because of the developmental stage of children up to 6 years; researchers similarly rated self-esteem as low because of the lack of a suitable measurement tool. The researchers had rated highest ‘not cooperating, throwing, spitting, won’t sit (challenging behaviour)’ on the basis of its impact on others and on the child’s experience. Both groups of health and education professionals identified a range of additional constructs that they would consider it important to measure, including communicative competence, problems with food, functional adaptive behaviour, etc. They also mentioned the importance of identifying the skill set of support staff, and parent confidence in managing their child’s needs and behaviours.

When all groups came together, the discussion highlighted differences in perspective, in summary a ‘social’ model and a ‘medical’ model. The parents and the young adult on the autism spectrum argued that it is important to focus on what children can do, to see autism as a ‘difference’ rather than always use a ‘deficit’ model, and to focus more on how to enable children through improving their environments. Parents were encouraged that the clinicians had mentioned including assessment of the skills of care and education staff. The clinicians reflected that their approach to assessment and intervention is based on a more ‘medical’ model: early identification of specific impairments, treatment, prevention of secondary impairment, and so on. The measurement of outcomes and tools available reflect this framework, with an emphasis on problems and deficits. For the researchers, the model of intervention and outcome assessment was also primarily embedded in a ‘deficit’ model of autism, with an emphasis on treating and measuring core features of autism. Research outcomes such as helping parents manage better and understand more are seen as ‘soft outcomes’, and not given the same importance as changing children’s characteristics. A certain contradiction was pointed out between the recognition that publicly funded research must now be informed by good patient and public involvement, and yet the priority research questions, commissioning briefs and frameworks for judgement of what is good science do not necessarily value the social model of understanding a condition such as ASD.