Inclusion criteria

The eligibility criteria for a participant group to take part in this aspect of the research were based on the child. They were that the participating child was:

• aged between 5 and 15 years
• attending mainstream school (this includes children who may have some mainstream lessons and some lessons in a ‘base’ or equivalent within a mainstream school)
• diagnosed with ASD by the multi-disciplinary, multi-agency York Autism Spectrum Disorders Forum or other equivalent body using the International Statistical Classification of Diseases, Tenth Edition (ICD-10)¹¹⁴ or DSM-IV²³ research diagnostic criteria
• exhibiting challenging behaviour as reported by parents and teachers. The definition of challenging behaviour we used was behaviour that those close to the child wanted to see decrease and which they believed could result in social exclusion.

Exclusion criteria

The following criteria detail the conditions which mean a participant group would not be included in the study.

• any child that had used Social Stories within the last 6 months was excluded
• if either the parent or the teacher had already taken part in one of the user groups (see Chapter 4), they were excluded from taking part to ensure maximum data saturation.

Goal-setting meeting

This was an initial meeting with participating adults to establish a simple goal for the intervention to focus on. The rationale for establishing this in the methodology of the feasibility study came from PPI representatives. They were keen to point out that Social Stories could be used for a very wide range of difficulties. This was also exemplified by the systematic review (see Chapter 2). They therefore wished to include an outcome measure that could capture this wide diversity. After taking advice from the Team Management Group (TMG), it was agreed that an individualised goal-based outcome measure would be included in the feasibility study. It was therefore decided that for each participant group, a meeting to establish and agree a clear goal would be necessary before measurement could take place. The pilot phase allowed the opportunity to explore possible ways of helping participants to set and rate goals at the beginning of the study and rate these goals after the intervention period.

A secondary purpose for the GSM was to examine the feasibility of operationalising two target behaviours related to this goal which, in a full-scale trial, could be assessed by an independent observer. It was hoped that the teachers and parents could provide enough information by meeting with a clinician and a trained researcher to operationalise these behaviours. If it proved successful an independent observer could then arrange a baseline observation shortly afterwards without encumbering participant flow excessively.

Goal-setting was initially conducted as a two-step process. Participating teachers and parents were asked to identify two goals separately (a goal focused on a positive behaviour and a goal focused on a negative behaviour) and secondly, rate both their own and each other’s goals. The pilot phase demonstrated that this process was cumbersome, time-consuming and resulted in lengthy delays in return of data. After consultation with the TMG, we changed this to a one-step joint goal-setting process. Participating teachers and parents were encouraged to jointly identify one goal they would like to work on during the intervention period. They were also asked to identify a positive behaviour and a negative behaviour associated with this goal. The mechanisms for measuring this and other outcome measures are described below.

Intervention

Adult participants were invited to the next available training day and were given a copy of the Social Stories training manual to read through. The training day used in this pilot lasted approximately 5 hours and aimed to teach how to create and deliver Social Stories to the participants. The first half of the session consisted of a presentation detailing the theory behind Social Stories and the practicalities of writing and delivering them. The structure of the presentation closely mirrored that of the manual, making reference to it in places. The second half of the training session was devoted to creating the Social Story for the goals set in the GSM for each child. Participating teachers had the central role in creating the story. Participating parents when present were there in an advisory capacity to offer insight into their child’s world. Guidance was also provided on how to format the story to individualise it to the particular child’s needs [e.g. story book format, Microsoft PowerPoint^® (Microsoft Corporation, Redmond, WA, USA) presentations or on a single sheet of A4 paper]. An example of an unformatted Social Story is provided in Appendix 10.

The intervention delivery period lasted 4 to 6 weeks. How often the intervention was delivered was left up to the participating teachers. This approach is consistent with the individualised nature of the Social Story intervention, which is driven by the child’s needs and ability. As a starting point, the researchers suggested that the intervention could be delivered three times a week for 2 weeks. However, adult participants were advised to use their judgement based on their knowledge of the child when deciding how much to present the story and for how long. Similarly, the participating teachers were not required to read the Social Story to children who would prefer to read it by themselves (as was the case with the older participants), as long as time was set aside for the child to have an opportunity to read it with the teacher present.

Qualitative interviews

Semi-structured interviews using topic guides were conducted with all participant groups after the intervention period. Interviews with participating parents lasted approximately 45 to 60 minutes. A short interview, ranging from 2 to 10 minutes, took place with the participating child, with their parent helping to facilitate the interview.

The participating teachers were also interviewed. The interviews aimed to elicit users’ thoughts and experiences of setting goals and completing the questionnaires. The interviews were an opportunity to get feedback on goal-setting, background information to the story, its implementation and impressions of how well it worked. The interview also considered the reading of the story and overall views on the presentation, structure and content of the manual.

Interviews were conducted by two qualitative researcher fellows, one acting as the main interviewer, and the second taking notes during the interview and picking up any outstanding questions or topics to probe. Following the interview a discussion between the two interviewers picked out the main issues and points in a recorded debrief. A brief two-page summary of the interview which included background information, brief comments and key points was made available to the writing team quickly after the interviews occurred. This made sure that up-to-date information was provided to allow iterative development of the training manual. Interviews were transcribed verbatim and imported into NVivo and thematically analysed.

During the intervention period teaching staff were also asked to complete a practitioner reflective diary (see Appendix 11). The diaries sought to capture information about facilitators and barriers to delivering Social Stories within existing service models and the participants’ perception of whether the intervention was or was not working well, including their views of why. Participants were asked to note experiences that highlight where Social Stories worked well and less well and to provide any perceptions of reasons for any discontinuation with the intervention.

Adult participants were encouraged to make any notes or suggestions on the manuals and these were collected in by the researchers at the end of the intervention period and fed into the writing groups to inform revisions to the manuals.

All participants were asked to complete outcome measure tools as designed for use in the feasibility trial. This included setting ‘goals’ and completing a questionnaire at 4 weeks post intervention start date. Space for comments and the opportunity to record the estimated questionnaire completion time was provided.

Outcome measures

The pilot provided the opportunity to pilot questionnaires to be used in the feasibility trial. Participants only completed the goal-based outcome measure at baseline. The full set of questionnaires (including the goal-based measure) were sent out 4 weeks after the intervention start date and consisted of:

1. The Strengths and Difficulties Questionnaire (SDQ).¹¹⁵ The SDQ is a brief child mental health screening questionnaire. It was completed by the adult participants and older participating children (11–15 years).
2. The Social Responsiveness Scale-2 (SRS-2).¹¹⁶ The SRS-2 identifies social impairment associated with ASD and quantifies its severity. It was completed by the participating teachers.
3. A goal-based outcome measure, designed by the research team (see Appendix 12) including operationalised behavioural measures. It was completed by all participants.
4. Bespoke resource-use questionnaires were developed by the health economist to capture the resource implications of the child’s behaviour at school and home see (see Appendix 13). These were completed by the adult participants.
5. The European Quality of Life-5 Dimensions (EQ-5D) proxy and the European Quality of Life-5 Dimensions youth questionnaire (EQ-5DY).¹¹⁷ These are standardised instruments for use as measures of generic health outcomes recommended by National Institute for Health and Care Excellence (NICE). They were completed by the participating parents and children respectively.
6. The Health Utilities Index 2 (HUI2).¹¹⁸ This is an alternative preference based generic health outcome measure to establish health states in children, and to report their health-related quality of life and produce utility scores. It was completed by the participating parents.
7. The Parental Stress Index (PSI-4).^119,120 The PSI-4 is designed to evaluate the magnitude of stress in the parent–child system. It was completed by the participating parents.
8. Spence Childhood Anxiety Scale (SCAS).¹²⁰ The SCAS is a 44-item questionnaire developed to assess the severity of anxiety symptoms broadly in line with the dimensions of anxiety disorder. It was completed by the children.

Timing of intervention

The timing of the intervention was seen as problematic for many of the adult participants. Delivering the intervention during the lead up to the Christmas holiday was a problem as this period is a particularly busy time with all the activities that occur. In some cases the intervention was squeezed or abandoned as a result, with parents and teachers feeling they did not deliver it as much as they would have liked. This could mean planning around school ‘pinch points’ such as statutory assessment tests (SATs) and holidays. Participating teachers suggested that it would be better to start the intervention early in a term, that is, in September or January.

Goal-setting

Throughout the pilot a number of issues and challenges with setting and rating goals were identified:

• A separate, two-step goal-setting process was not feasible.
• It became clear that goal-setting needed to be carefully guided by a clinician (defined as a mental health professional with experience of working with children with ASD and their families) to ensure that suitable goals were identified.
• Operationalising behaviours needed a large amount of time and the method needed refining.
• Joint goal-setting was not always possible or feasible.
• The goal identified at the outset did not always remain the goal of the Social Story actually written, or rated after 4 weeks of intervention delivery.
• The goal-setting process leads naturally to participants trying to identify ways to achieve this goal. This could have implications for accurately measuring the frequency of a behaviour at baseline, for example interventions/strategies could be implemented before a baseline measurement has been made.

Goal-setting was seen as important, as was having both adult participants present. Some people found that they were able to work on goals during training. The GSM provided a space for parents and teachers to meet and discuss the child’s difficulties. They expressed comfort in their ability to gather information and also share ideas in a protected environment. It was logistically difficult, however, to find times that were suitable to all parties. Providing people with a range of dates was considered to improve accessibility. PPI representatives were also clear after reviewing the feedback that the dates would also need to span different days and times.

One of the biggest issues beyond having people attending the GSM was developing measurable goals. Adult participants sometimes were overambitious in what they were attempting to set as their goal. In the meeting, people negotiated and limited the scope of the goal, with the aid of a clinician or someone from the research team. One participating teacher described this process as follows:

[The goal-setting was] very useful, . . . it helped, well just trying to unpick it gets you to the root of what it is that you’re trying to, trying to sort out, I think that was really good. It’s almost like walking down a path and clearing bits out, things that don’t really matter . . . Filtering and then realising that it’s, it’s really getting to the nitty-gritty and then seeing it much more clearly.

Female, teacher, primary school

A few people commented that the manual needs to include more guidance about goal-setting.

. . . really focus or, possibly a bit more training on that to assist [them].

Female, parent, primary school

Having access to members of the research team such as the consultant or researcher was useful in helping determine the goals, suggesting that the manual by itself was insufficient for some in helping with goal-setting. The manual was used more as a reference tool by some.

Yeah, yeah, I think it was more about the training, really useful to start off with as a kind of coming in, you have the training and then the manual’s a reference point to back it up . . .

Female, teaching assistant, primary school

Although happy with the goals set, one participating teacher found that they were very difficult to implement and measure. Another participant group made changes to better reflect the needs of the child. Adult participants recorded a dichotomy between achieving goals at school and at home which can sometimes mean that goals set in one place will not be as appropriate for the other.

We had an hour set aside so we [parent and teacher] talked through some of the issues at home, issues at school and then we chose a target we thought would be good for both home and school. I think in hindsight actually it’s probably a better target for home and, and the issue that we’ve picked hasn’t cropped up an awful lot. He’s been quite good at managing himself in a way, so if I’d had the time I’d have loved to have done a second story to target some of the classroom behaviour that he’s come back from, after Christmas.

Female, teacher, primary school

Training

All six participating teachers and four participating parents attended a training day. The training day was on the whole well-received. Interviewees reported it as being positive and informative. For some attending this was their first exposure to Social Stories. One participating teacher would have preferred a teacher-orientated training session that contained less introductory material on autism, as this had been covered as part of their teacher training. Others found this element enlightening. One suggestion was to shorten the length of the session when a teaching assistant had to get time off to attend. This was offset by concerns by some PPI members that a ‘good enough’ training session was needed to enable the best possible intervention construction.

One consequence of the training session was having quality time with knowledgeable people who understand Social Stories. This was seen as very important. This helped define goals further and shape the stories that were written on the day. It was considered that the training helped make sense of the manual.

I think both is good but if you couldn’t get the training the manual would help because it’s the same thing, but actually I think when someone explains something in person you get a better thing.

Female, parent, primary school

The manual seemed to supplement the training rather than the other way around and many reflected that the training and manual would need to be used together for best effect.

[We] went through the training and everything, and once they’d gone through parts of the manual we went into a separate room to try and write the stories, so, which, to start with, was very daunting to try and write it, because seeing everything on the screen and what was the, what was a descriptive sentence, what’s an affirmative sentence, and all this lot, just seeing the words didn’t really, I didn’t really get it until I actually saw some story.

Female, parent, secondary school

Additional areas were identified for further training, beyond understanding, developing, implementing and writing Social Stories. One teaching assistant would have liked to learn skills in using ‘communicate and print’. Two other people mentioned that support developing the physical story, that is adding visuals, using Microsoft PowerPoint presentations, fonts and layouts would be useful.

Manual

Participants were very thorough in their feedback about the manual. Beyond the interviews, a number of people had made detailed comments in the margins of their copy, which were passed to the writing group. These included points for clarification and areas for improvement.

Type of school

Primary school teachers reported an additional benefit from undertaking the pilot study in that they got to know their students better through the process of individualising the story and spending time with the child. They also reported an improved relationship with parents. This worked particularly well in primary schools.

The feedback from the participant groups in the secondary school was less positive about the study. The groups did not seem to have as effective a communication channel between parents and teachers working with the child in this setting. For example, one parent felt they had to ‘nudge the school’ to get them to take part. In addition, it was found that the structure of teaching is different in secondary schools, with one child often having multiple teaching assistants and less consistency of support from day to day. The secondary school teachers noted that it was difficult to implement Social Stories during the school day. They did not feel it was appropriate to use break times.

We were keen to learn from this point and reinforce to parents and teachers that a child should never be coerced or ‘forced’ to complete an intervention of this nature. This was seen as a learning point to reinforce the constructive and positive ethos of Social Stories.

Pupils at secondary school have a structured timetable there is little to no time to sit down with a pupil and really go through a social story. Pupils don’t tend to want to go through the stories in their break times and we feel that we can’t force them.

Female, teaching assistant, secondary school

Finding a quiet location without too many distractions was difficult, particularly in one school, as was finding time to make changes to the story. Teachers are very aware about potential stigma and discrimination with regard to the implementation of stories and considered carefully the best ways to implement them.

Teenagers are a lot more aware when they are different from their peers and therefore don’t like to stand out further.

Female, teaching assistant, secondary school

Goal-based questionnaire

In general, all the participants struggled with questions related to the frequency of behaviours in the goal-based outcome measure at both baseline and 4-week follow-up, finding it difficult to estimate the number of times the behaviour occurred. This may be down to the process of operationalising the behaviours in the GSM which did not result in behaviours that were readily measurable.

In three cases participating parents did not fully complete the goal rating and behaviour frequency section, and in two cases they commented that the goal and behaviours related to school so they were not able to answer the questions. In one case the parent changed the goal and behaviours and therefore rated a different goal to that identified at baseline.

One participating teacher left the entire goals section blank in the 4-week questionnaires relating to two pupils, noting that this section was confusing. Another left questions related to behaviour frequency and to educational outcomes blank. This gave us a large amount of information about possible necessary adjustments to these measures. PPI representatives gave invaluable advice subsequently in their redesign.

Parent questionnaire

Three participating parents commented that they thought some questions were irrelevant and inappropriate, or difficult to answer. In particular, one noted that the questions related to income and questions within the PSI-4¹¹⁹ were inappropriate. They also disliked a question referring to a ‘problem’. Two further parents left questions relating to income blank and also left questions with the health and community service use sections blank.

Child questionnaire

In two cases, participating parents reported that they completed the child questionnaire on behalf of their child, with both noting they felt the questions were unsuitable for their children, one particularly mentioned questions within the Spence Children’s Anxiety Scale.¹²⁰ One parent commented that asking the child the questions caused a lot of confusion. In one case, parents noted that they did not agree with the rating of ‘0’ their child had given the goal at 4-week follow-up.