Introduction
A critical component in the development and piloting of a new intervention is to determine what factors contribute to the overall success, or indeed failure, of the intervention.40 Process evaluation in trials of behaviour change interventions investigates the extent to which an intervention is delivered as it was intended.40,155,156 It is used to investigate whether or not the components of the intervention are delivered consistently and accurately to the target group.155,157 Process evaluation may also be used to monitor engagement with the intervention and to determine what individual-level factors influence engagement.158 Most importantly, careful process evaluation can help to identify: why an intervention was less successful than intended; what factors could have contributed to a successful intervention; and how further improvements could be made to the design or delivery of an intervention.159,160
Three essential dimensions for assessing the extent to which an intervention has been delivered as intended are fidelity, dose and reach.155 Fidelity measures the quality of delivery of the intervention, dose monitors how much of the intervention was delivered and how much was received by participants, and reach assesses the extent to which the intervention makes contact with or is received by the target group. Frameworks based on these dimensions have been created to evaluate studies and programmes, frequently with additional components including recruitment (of participants and agencies or personnel to deliver the intervention), context, barriers to implementation and maintenance (retention of participants).161–164
Process evaluation often involves interviews, focus groups or observational studies, with both study participants and those delivering the intervention.158 A challenge for non-contact interventions is to measure process without introducing personal contact into the study. This would change the nature of the intervention. There is a need to introduce new measures which do not interfere with the delivery of the intervention for non-contact interventions.
Research questions
To what extent were the text messages delivered as intended?
Were the messages opened, read and responded to by participants?
Did specific text messages successfully tap into key components of the behaviour change strategy?
Was the intervention acceptable to participants?
Methods
For this feasibility study, four methods of process evaluation were used. Fidelity of the delivery of the intervention was assessed from data captured by the computer system which monitored intervention delivery. It is possible to monitor receipt of text messages by individual telephone, which is equivalent to interviewing or observing those delivering an intervention. Engagement with the study was monitored by asking questions in text messages. The frequency of responses to these provided a measure of engagement with the intervention and assessed retention in the study. Content analysis of the responses given provided a method to ascertain the nature of engagement with components of the behaviour change intervention. This also identified parts of the intervention that were misunderstood or ambiguous. Finally, conventional process evaluation through interviews with participants at follow-up was undertaken to investigate the reported acceptability and impact of the study.
Research question 1: fidelity of the delivery of the intervention
The intervention comprised a combination of SMS messages (text messages) and MMS messages. The delivery was administered by a computer program at the Computing Department at the University of Dundee. SMS messages can be tracked to determine whether or not they were delivered to the telephone (it is not possible to track MMS messages). When SMS messages were not delivered to the telephone immediately, the computer program continued to try to send the message for 24 hours. If the message could not be delivered during this period, this was recorded as a delivery failure and the programme would then send the next message in the sequence.
Data captured on the delivery status of the SMS messages were recorded as: delivered (the telephone had reception and was switched on); undelivered (the telephone was switched off or it had no signal for 24 hours); or no status returned. The program did not record whether or not messages delivered to the telephones were opened. The proportion of SMS messages delivered to participants in the intervention and control groups was monitored as a measure of fidelity of the delivery of the intervention.
Research question 2: responses to text messages and characteristics of responders
Both the intervention and control packages included questions asked in text messages. The primary purpose of these questions was to promote engagement and to maximise retention of participants in the study. Responses to the text messages were received and collated by the Computing Department at the University of Dundee. The anonymised messages were screened daily by a member of the research team who was not involved in recruiting the participants or delivering the intervention. This ensured that any specific requests (e.g. to withdraw from the study, to find out where to get help for an alcohol problem, or a change in telephone number) could be dealt with.
On completion of the study, the collated data were analysed to gauge engagement and to examine the type of the men who responded by demographic characteristics and drinking history. The FAST121 and the Readiness to Change Questionnaire123 were used to measure hazardous drinking and attitudes to drinking behaviour. Intervention and control data were analysed separately. Study participants were categorised as high responders or low responders. High responders in the intervention group were men who replied to seven or more of the nine questions asked, whereas high responders in the control group were those who responded to at least three of the four questions asked.
Research question 3: content analysis of the responses received
Participants in the intervention group received a series of 36 SMS and MMS messages. These incorporated behaviour change techniques using social cognition models93 and MI124 which were organised by the stages of the TTM of behaviour change.165 Nine of the 36 messages requested a response to a specific question. The questions were carefully crafted to reinforce specific components of the intervention.
Control group participants received 34 SMS and MMS messages on four health topics: healthy eating, physical activity, sexual health and mental health. One message each week asked a multiple-choice question on the topic of the week. The control package was designed to provide information on each topic and to be interesting and funny as a means of keeping the men engaged. The content of the messages did not incorporate any behaviour change techniques.
Responses to the messages confirmed that the participants had: opened and read the message; understood the question; reflected on the content/context of the question; and given an appropriate and considered response. Analysis of content of the messages from the intervention group participants permitted an assessment of engagement with the study and with the psychological constructs of the intervention. Responses to the questions gave an indication of engagement with behavioural antecedents to reduced drinking:
recognition of reasons for drinking
awareness of the harms of heavy drinking
perceived benefits of drinking less
subjective norms (perception of significant others' beliefs about their drinking)
control beliefs (beliefs about factors that facilitate drinking less or impede drinking less)
perceived behavioural control (perceived ease or difficulty in drinking less)
behavioural intentions.
In addition, responses to the questions were used to determine whether or not the messages were understood and how they may be modified and improved for the full trial.
Research question 4: acceptability and impact of the intervention
Following the outcome assessment questionnaire which was completed by telephone interview, participants were asked a further 10 questions about their experience in taking part in the study. The questionnaire comprised both closed and open questions on the acceptability of the study, whether or not they had encountered any problems with the delivery of the intervention, and also if they had gained anything from taking part. Open questions encouraged men to reflect on what they may have learned, whether or not the intervention encouraged them to think about their health, whether or not they felt the need to improve their own health and any changes they were thinking about making or had made. This was designed to give insight into ways in which the intervention and control packages and their delivery could be improved.
Results
Research question 1: fidelity of the delivery of the intervention
The intervention package included 19 SMS messages while the control package included 14 messages. The MMS messages (17 for the intervention group and 20 for the control group) could not be tracked. A total of 1108 SMS messages were sent to the 67 participants during the intervention period. Of these, 1053 messages (95%) were delivered to the participants' telephones. Of the remaining 55 messages, 45 were recorded as undelivered (the telephone was switched off or it had no signal for 24 hours); no delivery status was recorded for the remaining 10 messages. Only 11 men failed to receive all of the messages, eight from the intervention group and three control group participants. The number of failed messages among these 11 participants ranged from 1 to 13, with a median five messages missed.
Research question 2: responses to text messages and characteristics of responders
Intervention group
Thirty of the 34 study participants (88%) in the intervention group responded to the text messages that asked questions (). More than half (18 men) replied to seven or more of the nine questions, with two replying to all nine questions and a further nine men answering eight questions. Of the four men who did not respond to any of the questions, two were also lost to follow-up.
Intervention group: participants' text message responses to the nine questions.
More than 82% of men in the intervention group responded to the first question (). For the remaining eight questions, an average of 20 men responded. Overall, there was little evidence of attenuation during the intervention period. In addition to replying to the questions posed, many of the men commented on other text messages. Nineteen of the 27 messages which did not request a response received at least one reply (see ). Responses to these messages simply acknowledged that the message was delivered, expressed empathy with the message, or responded to the humour in the message.
Intervention group: total number of responses to text messages.
Characteristics of the men in the intervention group who responded
A comparison of high and low responders was made to explore the factors which influenced engagement with the study. High responders [those who responded to seven or more of the questions posed (n = 17)] were more likely to be married or living with a partner (71% of high responders vs 29% of low responders) and were more likely to be in employment (77% vs 36%). More of the high responders were recruited from GPs than through RDS (71% vs 43%). The high responders were more likely to report that they had fewer (two or less) heavy binge drinking days (≥ 16 units in one session) in the previous 30 days (65% vs 43%). However, more of the high responders were at the pre-contemplation stage on the Readiness to Change Questionnaire (77% vs 57%).123 Factors that did not influence the frequency of responding included age, educational attainment, level of deprivation, frequency of binge drinking and FAST score.121
Control group
Thirty-one of the 33 participants (94%) from the control group responded to the text message questions (). One-third of the men answered all four questions and another third answered three of the four.
Control group: participants' text message responses to the four questions.
Fourteen of the text messages that did not seek a response elicited comments from participants (). The majority of comments on these texts were in response to humorous messages. More men answered the questions during the second half of the study, and 100% of the control group remained in the study at follow-up.
Control group: total number of responses to texts.
Characteristics of the men in the control group who responded
Comparing high and low responders (three or four responses vs two or fewer) showed that, like the intervention group, high responders were more likely to be married or living with a partner (73% vs 46%) and were more likely to be in employment (82% vs 63%). Fewer of the high responders (n = 22) were in the younger age group (27% vs 64%). More of the high responders were recruited from GPs (68% vs 27%). High responders were less likely to report binge drinking days (≥ 8 units in one session) on more than three occasions in the previous 30 days (50% vs 73%). More of the high responders were at the pre-contemplation stage on the Readiness to Change Questionnaire (64% vs 36%). Factors that did not influence the frequency of responding included educational attainment, level of deprivation, frequency of heavy binge drinking (≥ 16 units in one session) and FAST score.
Research question 3: content of the responses received
Intervention group
Engagement with the study was assessed through the nature of text messages received from participants. Responses to the questions monitored engagement with behavioural antecedents to reducing drinking.
Recognition of reasons for drinking
An early message (text number 4) was designed to identify the type of drinkers in the study, based on the DMQ,53 which categorises reasons for drinking as social, coping or enhancement. The message asked: ‘What's the main reason U drink? A It's a habit; B To feel better; C To have fun; D To cope; E To wash down your chips. Text me your answer’.
The majority of men (n = 23) indicated that they drink ‘to have fun’ and some gave more than one answer. Four men said that their drinking was ‘a habit’, while another five reported that they drink ‘to feel better’. Only one man said that he drank ‘to cope’. Two said none of the options applied, while one said he drank ‘to wash down his chips’. Two men proposed other reasons for drinking:
To unwind and help relax.
To have fun socialise and let my hair down. I work hard all week and when I get the chance I feel I deserve to enjoy my weekend.
Awareness of the harms of heavy drinking/perceived benefits of drinking less
Text number 9, ‘Can U think of any reasons why it may be a good idea for U to cut down a bit on your drinking? Text me your answer!’, was designed to encourage re-evaluation of current drinking behaviour. It was posed as a question so that the participant would not only voice an argument for change, but write it down in a text and return it to the researcher. Responses to the question fell into four categories.
Immediate benefits:
Not suffering with a hangover.
Feelin rough hangovers getin into silly situations getin into trouble all these things get u at some point with 2 much drinkin!
I wouldn't feel like crap in the morning, and my wallet would have more money in it!
I would want to cut down drinking to enjoy my night more and not forget parts of it. it would save on cash and avoid sore heads in the morning.
Health benefits:
Unhealthy and bad liver.
To stay healthier later in life.
For my health ageing of my skin.
Im putting on a lot of weight. And my health suffering as a result.
Family reasons:
To get fit and stay healthy for my family.
Live longer for my kids.
Financial benefits:
Money.
Save money no get hungover.
Text numbers 14 and 15 were delivered in quick succession, and were designed to encourage the participants to think about the pros and cons of changing their drinking patterns. The first simply asked ‘How much would U save every month if U drank half as much?’ The second, delivered 3 minutes later, said ‘Kerching! Kerching! Please count up your savings & text me the sum!’ Twenty-four of the 34 men answered the question. Their estimated savings ranged from £10 to £690 (two men said ≥ £600 and one said ‘400 quid easy’). Many of the responses indicated that the men had tried to do the calculation:
Hard 2 tell. But i would have at least saved 24 pound so far this week.
Im not sure but i think a good wee sum.
About between fifty and eighty pounds and thats a fact.
£200 a month or more easy. That would be on carry outs and the pub.
The next two text messages (text numbers 16 and 17) developed this theme by asking how saving money could enable them to buy items they wanted. This transforms potential benefits from the abstract to concrete. The first message stated ‘By saving up your cash U could treat yourself to something special!’ The second, delivered 3 minutes later, said ‘Try to picture what U would like to buy & text me back your answer!’ The men easily identified a range of ways to spend the saved money from simple treats to extravagant holidays. Responses demonstrated that the men had thought carefully:
Saving that money would help me take my girlfriend out for a meal now and then.
Definitely a car possibly a few more holidays – love buying designer clothes.
Xbox 360 with Kinect.
A 3d television that would be good or a nice holiday in the sun braw.
Trek 2.5 road bike – cost £1650.00 RR.
Holiday to Australia for 3 weeks.
One text message was a direct quote taken from a focus group: ‘Andy from Dundee says – “I cut back on my drinking because my father in law died of it” What would be a good reason for U to cut back? Text me back!’ It elicited deeply personal responses on the long-term benefits of reducing:
I really wanna stay out of trouble and not become the person I can be after a few too many.
I would b able to make the most of the next day rather than feeling shite.
Ive have tryed because i seen my dad nearly die.
Good reason for me cutting back was again, looking after my son. I can't allow drinking to interfere with my job either. My Grandad was an alcoholic, so I know the health risks associated with booze.
So i,m able to be here for my kids to grow up and have kids them self.
For my health so I can prolong my life and spend time with my family and money which could be better used on other things.
Subjective norms
Text number 22 asked ‘Can U think of someone who'd be happy if you made a change! What would U hear them say? Please text me your answer?’ This text message tried to get participants to identify people who would approve of their decision to reduce their alcohol consumption, to increase their self-efficacy about making a decision to change. This question elicited deeply personal responses. Parents, partners, family members and friends were identified as people who would be pleased to see a reduction in drinking. Some men gave detailed responses on what their family and friends would say:
Thats brilliant what u have done,maybe we can do something at the weekend.
yes my friends & family would say well done & good on you keep it up & stay focused & positive abou life because u only get 1 chance.
My partner. She would be happy for health reasons. That i would maybe lose weight.
Two of the men gave a light-hearted but nevertheless thoughtful response:
They would say what a peaceful night not having to deal with a drunken ass.
My dad. Its good ur no phoning me for a lift at 2am!
Only one man did not give a positive response to this question:
She drinks more than me so she will probably ask if I want a drink.
Control beliefs/perceived behavioural control
Text number 32, the final question in the intervention package, was a multiple-choice question which was intended to both motivate and challenge the participants. It was also designed to encourage the belief that change is possible. The message stated, ‘Many people find it easy to reduce their drinking. Do U think U could if U tried? A Yes; B No; C Maybe. Please text me your answer!’ This question was given an overwhelming positive response by the men who responded. Seventeen of the 21 who responded answered yes, with one man saying ‘A,,,, for sure’. Four men said ‘maybe’ and none said ‘no’.
Behavioural intentions
One of the last messages (text number 34) stated ‘Liver disease (cirrhosis) is the major cause of death in heavy drinkers. Drinking less will greatly reduce your risk of liver failure.’ One spontaneous response to this message was:
Im away 2 try and cut down or stop from monday.
Identifying ways to improve the intervention
Ambiguous question
One question was misinterpreted by approximately half of the men who responded, text number 28, ‘Can U think of any obstacles or barriers that stop U drinking a bit less each week? Text me your answer!’ Although the question asked for barriers to reducing alcohol consumption, some men listed factors which facilitated drinking less:
Work goin someplace in the car. Kids commitments.
Driving and work and playing football and definitely when I look after my daughter!
yes Driving.
Prices going up.
Yea money,, getting up early for work with hangover,, and prices in pubs,,,, wow.
Eight of the 12 men who appeared to interpret the question correctly felt there were no real barriers if the individual is committed to changing:
There is nothing to stop me drink less if that was what I wanted.
there isn't much that can stop me apart from me myself. anything else is just an excuse
I have no obstacles or barriers that stop me drinking a bit less each week, it's more of a lifestyle choice for me, and the fact I'm raising my son on my own.
Some men, however, were able to identify barriers:
I dont always know when iv had enough.
My friends asking me to meet them in the pub for a couple.
Boredom habit stress.
The question was effective in encouraging the men to identify barriers, so it should be retained in the full trial, but should be re-phrased to be more easily understood.
Unpopular question
One question proved unpopular, with only seven men answering it. Text number 27, ‘How much did U spend on alcohol this week – Please text me your answer!’, incorporated the image of a man with no money in his pockets and a banner saying ‘SKINT!’ This message was designed to make the participants think about the pros and cons of drinking, and to highlight the negative consequences. The amount spent ranged from zero to:
55 quid I reckon which isn't bad! I think;-)
Ive spent seventy six pounds this week ok mate
Those who responded gave appropriate answers, so it is unlikely that the question was misunderstood. The men had already answered questions that addressed how much money they could save if they drank less. Thus, the low response may suggest that participants felt the topic had already been covered. Most studies repeat messages given in an intervention as a means of reinforcing the intervention.23 This message contained one of the few loss-framed questions, and indicates that this approach should be avoided in the full trial.
Control group
All of the questions posed to the control group had multiple-choice answers (e.g. what proportion of Scottish men are overweight or obese? Which is the most common sexually transmitted disease?) The correct answers were sent to the men 2 hours later. The majority of responses received were single letter (A–D), but some men wrote out the answers in full. Responses to the other text messages were typically a comment on a humorous text message.
Research question 4: follow-up questionnaire
The follow-up questionnaire was completed by telephone interview 3 months after recruitment to the study. Of the 67 men recruited, 64 (96%) completed the study. The three men lost to follow-up belonged to the intervention group. This section describes reported access to the text messages and reported acceptability and impact of the intervention. The results of the impact of the intervention on the primary and secondary outcome measures are given in Chapter 9.
Access to the intervention
Ten men reported that they had problems opening some of the messages (). However, all of the participants managed to resolve the problems. Four men, whose telephones could not receive images, were given new, low-cost mobile telephones. The remainder managed to open the images through the internet on a personal computer or contacted the service provider to find out how to open images on their telephones.
Acceptability of the intervention
Acceptability and impact of the intervention
More than one third of the men reported that they would have liked more text messages. None would have wanted to receive fewer messages (see ). Only one man reported that the timing of the messages could be inconvenient (while he was at work).
Willingness to share text messages with family and friends was used as a marker of acceptability of the intervention. The majority of the participants (95%), from the intervention and control groups, told other people about the study. More than 40% had discussed the study with more than five people and more than a quarter of participants had forwarded texts to friends and family members.
Almost all participants (94% from both groups) felt that taking part in the study was worthwhile, and when asked to give a score out of 10 for the study all said ≥ 6, with more than one third giving a score of 9 or 10 out of 10. The majority of men from both groups reported that they benefited from taking part in the study. More than 90% of participants from both groups said that they found the study informative (). The majority (86%) were encouraged to think about their health, and half of the men said that they wanted to make changes to improve their health. There was no statistically significant difference between the intervention and control groups in responses to any of the questions. Almost 80% of participants from both groups reported that the messages made them want to make changes to improve their health. In addition, just over half (58%) of the intervention group and 42% of the control group reported that the study helped them to reduce the amount they drank.
Impact of the intervention
Discussion
This feasibility study has shown that the client group (disadvantaged young and middle-aged men) are very willing to actively engage in a study with an interactive intervention. Through careful process evaluation the study has shown that participants had not only received, opened and read the messages, but thought deeply about the content and had taken the time to respond. Many of the men gave carefully considered personal responses to the questions set. The study has also shown that interest in the intervention was maintained for the duration of the study period. There was very little attenuation in the number of men responding to text messages, with > 60% replying to the final question.
Fidelity of delivery of the intervention is crucial.155 An important advantage of interventions delivered by mobile telephone is that the fidelity of delivery can be accurately measured. The computer system which delivered the SMS messages recorded that almost all (95%) of the messages were successfully delivered to the telephones of the participants. The high number of responses to the intervention text messages implies that the men opened all of the messages, confirming that exposure to the intervention was comprehensive.
Bellg et al.166 purport that measuring fidelity of delivery goes beyond ensuring that the content of the intervention is constant and that those delivering it are trained. Participants should understand the information given, particularly if literacy levels are low. This study monitored comprehension of the messages by assessing the nature of responses to the questions. The men clearly understood the language used. This was confirmed in interviews at the end of the study with one man reporting that ‘Anybody could have understood the messages even if you were dyslexic’.
A key finding is the value of content analysis of the responses to questions asked. This provides a further important dimension of fidelity that has not been previously reported.23 There are several advantages of using this methodology. Critically, the process evaluation does not alter the delivery of the intervention. The study was designed to have a non-contact intervention. Obtaining process data through interview or focus groups would have altered the characteristics of the intervention. Another attraction of this method is that it is unobtrusive. The questions were embedded in a series of non-threatening text messages, which participants could choose to ignore, with no likelihood of being pressed for an answer. The result was that personal questions were asked in a way which elicited deeply personal and apparently honest answers.
Answering questions by text message allowed participants to give a considered response to the questions asked, a luxury not afforded during a face-to-face interview. This incorporates a technique used in MI, and is particularly apposite for the questions asked in this study. Participants were asked to reflect on aspects of their drinking behaviour (e.g. reasons for trying to reduce consumption), identify who would be pleased with changes made and identify potential financial benefits of reduced consumption. Reflecting on these questions, formulating and writing down the answers, reinforced the messages. While this was an integral part of the intervention, it also served as a valuable tool for process evaluation.
A notable feature of the responses to text questions is that there was a high level of engagement from the start of the study with little sign of attenuation during the course of the intervention. Another early indication that the men were willing to engage in a narrative was in responses to multiple-choice questions. Very few men gave the one-letter multiple-choice answer: instead they transcribed the whole answer (e.g. in response to the question on reasons for drinking a typical reply was ‘C – to have fun’). This raises the question of why men were so willing to respond to the text questions. It may be because text messages are viewed as part of a conversation to which responses are expected.167 Certainly the high response indicates that men were comfortable with the study.
For this study, this dialogue was initiated by the process of obtaining consent by text message. Thus, a relationship was established, and the source was seen to be credible, an important factor in engagement.168 This high success may have been achieved because one entry criterion for the study was that participants should have access to a mobile telephone for the duration of the study.
All process evaluation components were also measured in the control group. Control group participants were also fully engaged with the study, with 100% retention at follow-up. This has important implications for the full trial. The control group package must be interesting enough to keep participants engaged in order to avoid differential loss from the control group during the trial. The messages were designed to give health information only, but in a way that did not antagonise the participants. Further, to avoid bias, the control participants should be given a similar amount of attention and support as the intervention participants.169,170 The text questions provide this attention.
Conclusion
The interactive nature of the intervention was successful in engaging participants, both in the intervention and control arms of the study. This feasibility study has identified a new and very useful tool for process evaluation in assessing complex interventions delivered electronically. Content analysis of responses to text messages not only confirms fidelity of the delivery of the intervention. It measures the extent of engagement with components of the behaviour change strategy, identifies ambiguity in messages and highlights gaps in the intervention and areas for improvement. Retention in the study, at 96%, confirmed that the overall design was acceptable to the target group.
