The training package consisted of a 2-day workshop plus 2 additional follow-up days. These were delivered over a 3- or 4-month period, allowing facilitators time to implement the care strategy in their service and to practise delivering the review to stroke survivors. Prior to commencing recruitment to the trial, the progress of the facilitators needs be checked to ensure that they understand the care strategy and are delivering it as intended.
The nature of the care strategy is such that it is flexible and emergent. This means that assessment of progress needs to explore whether or not the individuals working as facilitators understand the key features, the key purposes and the key activities of the care strategy; we also need to know whether or not facilitators are able to apply these in practice. The resources available to the LoTS2Care research team, alongside the geographical spread of the sites, mean that observation of delivery of the New Start review has not been possible for all facilitators at all sites. Moreover, it is recognised that observation of one review provides a ‘snapshot’ of a facilitator’s understanding and ability to apply this. For these reasons, the approach to assessing the progress of facilitators uses multiple components. These are outlined in the following sections.
Observation (when possible)
If possible, observing delivery of the care strategy is beneficial because it allows the researcher to see whether or not facilitators are able to apply their knowledge and skills of the New Start care strategy in practice. Natasha Hardicre (research fellow leading on training and implementation of the New Start care strategy) observed delivery of the review in the facilitators’ services [these observations took place during the time between the initial training course (September 2016) and commencement of recruitment to the trial (January 2017)]. The benefit of this method is that the review was done in its ‘natural’ setting. This allowed other factors to be observed, such as the ways in which facilitators and stroke survivors interacted with the environment and the interactions that occurred before and after the review, for example doorstep interactions or navigating clinical settings. Natasha observed four reviews in three services, delivered by three facilitators. Three of these reviews took place in a clinic within a hospital setting and one took place at the home of the stroke survivor participating in the review. All stroke survivors gave their verbal consent that they were happy for someone to observe their review.
An observation guide was created to provide observers with a structured framework when observing a review. This guide draws on the logic model. It contains sections to report what has been observed and a section where feedback can be noted down.
Oral or written test
Observation, although helpful, can provide only limited insight into a facilitator’s understanding of the care strategy, and their ability to apply key principles in that particular situation. Therefore, testing understanding through an oral interview or written test will be used to supplement any observation that is done, or as the primary means of assessing knowledge and understanding if observation is not possible. Specific questions will be asked to explore the facilitators’ understanding of the key features of the New Start care strategy, the differences between New Start and their previous/other ways of providing reviews to stroke survivors, the key purposes of the care strategy and how these are achieved, and the activities associated with their role as a facilitator. There are a number of ways of administering these questions.
They can be given as a written test for facilitators to complete and return. The advantage of this method is that facilitators have an opportunity to reflect on what they have learnt and spend time articulating their answers in ways that reflect their knowledge and understanding. There is, however, some risk that facilitators will rely too heavily on their notes or other resources, resulting in a presentation of their knowledge that does not reflect their actual understanding of the care strategy.
Moreover, facilitators need to be familiar enough with the care strategy and the materials to be able to deliver it effectively and flexibly with stroke survivors and their carers. Alternatively, then, facilitators could complete this under ‘test conditions’ to examine the degree to which they can accurately recall what they know and understand about New Start. However, sitting a test under exam conditions can be a cause of anxiety and stress for some people, and this may inhibit their ability to recall information. It may not, therefore, accurately reflect their ability to actually deliver the care strategy to stroke survivors.
Another alternative is to interview facilitators and ask the questions verbally. This would enable the facilitators to describe their understanding of the care strategy in a conversation. This enables the researcher to prompt or probe the facilitators in order to explore their understanding and go beyond surface-level description. The conversation could be recorded and transcribed for records or further analysis if required in the future. Oral interviews provide a way of checking that a facilitator is familiar enough with the care strategy that they are able to recall its key features, while allowing facilitators the time and opportunity to describe them through the course of a conversation. The researcher administering the interview can also seek clarification about points they are unsure about and this can provide a more robust and comprehensive assessment of progress and competency. It was agreed by the Trial Management Group that interviews would be carried out via telephone between 11 and 20 January 2017. Louisa Burton (research fellow) carried out these interviews. The Trial Management Group agreed that, if assessments were needed in a future trial, they would be happy for the questions to be administered and completed via interview in person or by telephone, or as a written test.
Reflective reports
Reflective practice is something that the facilitators are being asked to engage in on a regular basis. At the training course, weekly reflection was suggested as good practice, although reflective reports are requested only monthly so as to reduce burden on facilitators. Facilitators are able to choose what they reflect on for these monthly reports, although guidance was given at the training workshop (e.g. implementation issues, occasions when reviews went well, occasions when reviews were challenging). Facilitators are also able to choose how to present their reflections (e.g. visually, structured report, diary/journal entry).
As part of the assessment of progress of the New Start facilitators, they are being asked to complete two additional structured reflective reports. Each of these focuses on an occasion when the facilitators have delivered the New Start review. The report asks facilitators to describe the situation, reflect on their experience, compare it with their expectations and apply their theoretical knowledge, and suggest ways that they could learn from the experience. Facilitators were sent these reports on Thursday 5 January 2017; the deadline for completion and return was Wednesday 18 January 2017.
Assessing competency
Assessing progress is part of the training package, but, for the purposes of the trial, it is also important to establish whether or not facilitators are competent at delivering the intervention as intended. To use these means of assessment, structured guides have been created to allow a researcher to conduct the assessment and decide whether or not a facilitator is ‘competent’. Possible outcomes of the assessment are:
Facilitator is considered ‘competent’ – they have sufficient knowledge and understanding of the key features, aims and activities of the New Start care strategy, and they can demonstrate an ability to apply their knowledge in practice.
Facilitator is not considered ‘competent’ yet – if a facilitator does not yet have sufficient knowledge and understanding of the New Start review, or has not been able to demonstrate an ability to apply their knowledge in practice (e.g. insufficient practice at delivering it), a training plan will be put in place. Reassessment will then take place at a future time point.
Reassessing competency
Reassessment will use the same components as the original assessment: assessment questions and review of a reflective reports and activity record. It is expected that the facilitator will deliver at least one review between the original assessment and reassessment; this will give them an opportunity to apply any additional knowledge or understanding that they have gained. It will also provide them with an experience to reflect on. Only one additional reflective report will be required for reassessment.
