Objectives
The overall objective of the qualitative research was to understand the influence of modelling, including the use of machine learning techniques, in the context of the national audit to support efforts to maximise the appropriate use of thrombolysis and reduce unnecessary variation.
Specifically, the aims were to:
explore current understanding and rationale for the use of thrombolysis for ischaemic stroke to establish reasons for the variance in the use and speed of thrombolysis
understand physician perspectives on clinical pathway simulation and machine learning feedback to influence how simulation can be incorporated into
SSNAP to have a positive impact on practice
identify potential routes for the implementation of machine learning feedback to inform and improve future stroke management
explore how physicians interpret the potential consequences of following changes in pathway suggested by simulation.
Qualitative research was conducted in accordance with guidance offered by Blaxter.43
Intended data collection
Two experienced qualitative researchers (JF and KL) developed qualitative data collection and analysis protocols (see appendix). We planned to collect data through individual and group interviews with physicians involved in stroke care. We intended to conduct seven individual and three group face-to-face pilot interviews locally to determine variance in clinician approaches and attitudes to the management of ischaemic stroke and thrombolysis practice. We wanted to identify and pilot our topic guide and a range of visual displays and other methods of feedback from both clinical pathway simulation and machine learning for use in subsequent interviews. We then planned to undertake individual and group interviews, in person, with 20–30 career and training physicians.
To maximise participation in this research, we aimed to recruit physicians via the National Institute for Health and Care Research (NIHR) Clinical Research Network’s Stroke Network. To be eligible for inclusion, physicians needed to be employed at an NHS hospital and be involved in delivering stroke treatment at the time of the interview. Our sampling frame was premised on thrombolysis rates at the participating NHS hospital, which we divided into tertiles [i.e. low (8.9% and below), medium (9–13.9%) and high (14% and above)], as well as physician specialism, physician grade and door-to-needle time.
We planned to conduct interviews with 30–40 physicians and to conduct a follow-up stakeholder workshop with approximately 30 people. From the SSNAP data, as of 2018, we identified 20 potential sites for recruiting physicians, with six to eight sites in each of the tertiles for thrombolysis. Ethics approval was provided by the HRA and Health and Care Research Wales (reference 19/HRA/5796).
We worked with NIHR clinical research nurses and/or research and development (R&D) leads local to each site to identify a physician, nurse or R&D lead to act as a principal investigator, through whom the clinical research nurse would facilitate recruitment of physicians. This procedure was more complex than we could have anticipated, with each site operating with a different configuration of R&D personnel and requiring different variants of paperwork – despite our adherence to all requirements required by HRA ethics approval (e.g. the Organisation Information Document for Non-Commercially Sponsored Studies). This was a lengthy process, which led to us commencing recruitment at the same time as the first and second lockdowns for COVID-19. This significantly diminished the pool of physicians from which we were able to recruit, as physicians were redeployed to front-line duties and we were not a priority study for recruitment. In addition, we required a further amendment, for HRA approval, for us to conduct all interviews remotely. Preparation for a third HRA amendment, which would have identified 10 further sites for recruitment, coincided with the third lockdown for COVID-19, and most Clinical Research Networks and NHS trusts that we approached told us that they did not have the capacity or capability to support the study.
Actual data collection
To pilot our interview approach, we undertook a face-to-face group interview with a small group of medical registrars on a regional rotation that included stroke in their clinical setting. Having given written consent to interview, a senior modeller (MA) provided the registrars with a demonstration of the modelling process and outcomes, prior to the qualitative researchers piloting the topic guide. Feedback from the stroke physicians suggested that this approach was appropriate and produced data that were fit for purpose.
Our approach was subsequently modified for remote delivery, with all interviews conducted via Teams (Microsoft Corporation, Redmond, WA, USA), Skype™ (Microsoft Corporation, Redmond, WA, USA) or Zoom (Zoom Video Communications, San Jose, CA, USA), depending on the medium that was allowed in each NHS trust. At the beginning of each interview, participants watched a 10-minute video made by the senior modeller that contained examples of the process and outcomes of the machine modelling and pathway analysis, as a stimulus for discussion.44 The topic guide was then used to elicit participants’ own experiences of thrombolysis and perspectives on machine learning, alongside observations of group interactions and clinical settings.45
During the interviews, we collected data about physicians’ backgrounds, their attitudes to thrombolysis and their understanding of variance, their perspectives on machine learning and potential loci for the implementation of machine learning feedback (within and beyond SSNAP), and established the physicians’ views on possible unintended consequences that may result from changing the acute stroke pathway and potential means of mitigation. Our fieldnotes reflected the challenges of conducting interviews via video, with physicians often in clinical settings and sometimes wearing personal protective equipment, as well as capturing the dynamics between physicians who were working remotely from each other.46,47
Towards the end of the project, and during the third lockdown for COVID-19, we undertook an online discussion of our results with a small group of physicians (n = 3) who were identified via an annual meeting for trainees, organised by the British Association for Stroke Physicians (London, UK). The modellers (MA and CJ) presented a further set of outputs from their analyses, and the discussion focused on how additional modelling outputs might be used to facilitate quality improvement and inform service delivery.
The number of participants recruited was smaller than originally planned because of the NHS focus on coping with the COVID-19 pandemic and because of the challenge in accessing staff from units with lower thrombolysis use. The interview participants were also skewed heavily towards medical practitioners. In any future study we recommend having more qualitative resources to help recruit more staff, especially from units with lower thrombolysis use, and to broaden the interview base to include more non-medical practitioner staff (e.g. specialist stroke nurses).
Data analysis
Interview data were transcribed by an independent General Data Protection Regulation-compliant transcriber, and fieldnotes were written up by the two researchers. All data were anonymised and managed in NVivo for Teams (QSR International, Warrington, UK). Both researchers read all the transcripts to develop preliminary ideas and understanding. We developed these ideas alongside further re-reading of the transcripts, using a framework analysis aligned with the four broad exploratory objectives of the study, but, crucially, with an openness to any new insights from the physicians.48,49 Analytical summaries across multiple cases were created independently by both researchers and were used to explore the data. We held repeat discussions to develop the analysis, looking for negative cases and resolving differences of opinion about interpretation.50 In this way, we were able to examine these physicians’ accounts of their use of thrombolysis and orientation to machine learning and simulation. As our analyses developed, we also discussed our findings with members of the wider research team.
Results
Summary of key findings
Qualitative research demonstrated a varying openness to machine learning and modelling techniques:
Broadly, those units with higher thrombolysis use engaged more positively with the research and units with lower thrombolysis use were more cautious.
Clinicians from units with lower thrombolysis use tended to emphasise differences in their patients as the reason for lower thrombolysis use. Clinicians in units with a middling use of thrombolysis tended to emphasise access to specialist resources as being key in being able to deliver thrombolysis well. Clinicians in units with higher thrombolysis use tended to emphasise the work and investment that had gone into establishing a good thrombolysis pathway.
Clinicians wanted to see the machine learning models expanded to predict probability of good outcome and adverse effects of thrombolysis.
Despite this being a small study, physicians engaged with the machine learning process and outcomes, suggesting ways in which the outputs could be modified for feedback to stroke centres and utilised to inform thrombolytic decision-making.
Interview participants
We recruited 19 participants who took part in three individual and five group interviews (). Fourteen participants were consultants (specialising in stroke, neurology or elderly care), four were stroke registrars and one was a specialist stroke nurse. Ten participants were female and nine were male.
Details of interview participants, including gender
Current attitudes to thrombolysis use
Differences due to patient characteristics
Physicians working in hospitals with lower thrombolysis rates were more likely to suggest that a significant barrier to thrombolysis was the delayed presentation of patients, which could be magnified by suboptimal ambulance services:
A lot of patients present outside the window of thrombolysis at the hospital.
Site B
I think we rarely hit the 11% per cent national numbers, probably because patients come just outside the thrombolysis window.
Site A
Those physicians working in hospitals with lower thrombolysis rates were more likely to report that their patients were ‘different’ from those presenting at other centres, in terms of rurality, ethnicity, frailty and socio-demographic factors:
We’ve a slightly older population . . . we’ve slightly more bleeds than infarcts . . . we’re a slightly larger geographical area, so sometimes people are a bit delayed getting to hospital and we operate across two sites as well.
Site C
The above physician also highlighted that, because of these complexities, decision-making about thrombolysis was the most difficult part of their job.
Although population differences were also acknowledged by physicians at higher-thrombolysing centres, they were more likely to articulate the centrality of patient heterogeneity in their decision-making:
Consultant 1 (site H):
I wouldn’t be giving thrombolysis for various reasons . . . They’re often late, or got a very mild deficit, or they’ve got something that makes you feel extra wary about treating them . . . we’ve got a population that’s increasingly frail, they’ve got multiple comorbidities . . . [but] every patient is unique.
Consultant 2 (site H):
We all have different approaches, I say to myself the first question is, if I don’t thrombolyse this patient, what is the worst neurological outcome they could have? What is the disability going to be? And then the next question is how far are we down the time pathway, what’s the risk of bleeding here? And then, what are the little things that feed into pros and cons, how does that alter the equation from a standard patient? Is it that the benefits are going to outweigh the risks, how finely balanced is that decision?
Consultant 3 (site H):
The days of people being textbook strokes are long gone . . . we don’t see them . . . we don’t have a blanket policy. We eyeball them. And if they look dodgy, we park them and work out what’s going on, if they don’t look dodgy, we go straight to the scanner.
Differences due to differences in specialist resources available
Interviewees in units with a middling use of thrombolysis suggested that some of the delays in patient presentation could be mitigated through treatment by stroke physicians, rather than generalists, or by the involvement of a specialist stroke nurse:
We typically have a more deprived population, so accessing health care and time to hospital [and] our ambulance service is not as good . . ., a burden of disease due to deprivation . . . we do see a lot of young strokes . . . smoking, drinking, drug abuse . . . expertise is important there, so if you looked at our patients . . . the ones that had been given thrombolysis under 30 minutes . . . nearly all of them had been managed by a stroke registrar or a geriatrics registrar or a geriatrics consultant.
Site E
Stroke nurses being there increases the speed . . .
Site D
Those physicians currently working in centres with low or medium thrombolysis rates seemed more likely to emphasise the equipment that they lacked and that they perceived would improve the accuracy and speed of their decision-making.
Similarly, physicians working in hospitals with higher thrombolysis rates suggested that their higher rates were because of access to scans and other specialist facilities, as well as 24-hour stroke services:
We’re a big teaching hospital . . . that’s also got a trauma centre.
Site H
Thrombolysis is done by registrars with consultation on the phone with some access to the imaging for the consultant . . . there is no dedicated stroke team at night . . . we have a big variation between out-of-hours and in-hours door-to-needle time . . . it’s 38 minutes, out of hours it’s 89 minutes . . . don’t thrombolyse wake up stroke . . . MRI [magnetic resonance imaging] . . . perfusion scan . . . we don’t have the facilities.
Site F
On SSNAP data, we are one of the top-performing units in the country and that has happened through years of planning and hard work, where we take direct admissions, 24/7, we don’t do remote assessments . . . it’s always face-to-face assessments by consultant . . . with a specialist nurse, to see a patient, etc. And we have access to scans directly, including vascular imaging . . .
Site G
Therefore, the provision of more diagnostic tools was perceived as enabling a more nuanced approach to risk management, one that went beyond tallying risk factors and individualised patient care for more ‘marginal’ cases:
If I might manage a level of uncertainty about the onset time and some other characteristics, medications, for example, a slightly imperfect history that I have, if it’s a very severe stroke, it’s going to be a disabling stroke and I feel that the risks are outweighed by the benefits . . . I think that stroke severity and my perception of the ability to benefit from thrombolysis will then weigh into how much uncertainty I’m able to cope with, with the other things.
Site G
Although the sample interviewed was small, they were diverse in their attitudes to thrombolysis use.
Perspectives on simulation and machine learning
Acceptance of machine learning
Physicians who identified as confident thrombolysers had an initial scepticism of both the premise and methods employed in the simulations they were shown, although this scepticism was later dispelled:
The first thought that came to mind [with the modelling] was an innate assumption that doing more thrombolysis is a good thing . . . So, your machine learning may tell us how to do a lot of people who possibly don’t need it, possibly. I’m not saying that’s necessarily what you’re going to do, but it’s where it might go if we just say ‘more is better’.
Site H
Physicians who both worked alone and were interviewed in isolation were more anxious about how the simulation might be used to hold them to account for their decision-making and identified perceived risks:
I’d be suspicious if such a tool was available and a patient wasn’t given thrombolysis, then that might involve the lawyers and the legal teams.
Site C
I think safety would be the top thing, isn’t it, it’s got to be a hundred per cent safe and I think if you are close to a hundred per cent safe, if you can show that, if you can show that it’s safe and it doesn’t cause any negative outcomes for patients, then – and it also enhances patient care by speeding the process up, then I think you’ve won. If there’s doubts about its safety, even if it does speed things up, people aren’t going to trust it . . . clinicians are always wary about litigation, as well . . . some of this software could be used retrospectively . . . it could lead to decision-making being criticised retrospectively.
Site E
Those interviewees who had the benefit of working in a team with both a culture of collaboration and professional challenge were more inclined to see machine learning as a resource to draw on for their own decision-making. For example, interviewees in low-thrombolysing centres suggested that it might augment their decision-making, whereas interviewees in high-thrombolysing centres viewed it as a positive challenge to inherent assumptions that they might have developed:
I think it would be a help if there was a patient where, you know, maybe somebody else would have given thrombolysis them and we might see something that we weren’t doing that we would then, you know, implement as an action, if there was something clearly that we could be doing, you know, that would improve the rates.
Site B
. . . if you have a computer model you might get out these things out of, well, at least you think about it if the computer says something, then you have to have a strong kind of argument to refuse, to say no [laughs]. To say, well, the computer says, well the modelling comments that they should be given thrombolysis, why do you say it’s not given thrombolysis, you can’t just say oh, because he’s old or whatever.
Site F
It would be useful to know what would somebody else do. Now, whether that’s presented as a number or as a likelihood for thrombolysis . . . the hospitals . . .
Site F
In addition, those in the highest-thrombolysing centre also thought that the modelling outputs could extend their quality improvement initiatives:
It’s just a tool, isn’t it, it’s just another tool. We would never – you’d never base your decision on what the machine said! I mean, not until it’s like, you know, the Star Trek computer! . . . You’re generating data for improving a process and for understanding of process, so it’s very helpful for that . . . And it might be useful to beat the managers and say we need help with this, that and the other, but then any audit does that.
Site H
Suggestions for additional data to collect
Some participants suggested additional variables that they would like to see included in the modelling:
There are factors there which we would use in our decision-making process which are not listed as inputs . . . active bleeding, head injuries, blood pressure, whether the patient assents or consents [inputs]are insufficient and superficial . . .
Site G
The other thing that feeds in is that not everyone’s comfortable looking at CT [computerised tomography] heads and some people are waiting for that to be reported and I think that can add considerable time . . . especially down here, the radiology registrar is not always based in this hospital. So, they cover the whole of the [area] and they might be based in [other centre], whereas in the daytime you can just walk round the corner and speak to the radiologist reporting the scan, and say ‘What do you think, is it OK?’, or call the consultant on call. But I think if you’re, for example, a med reg in another speciality thrombolysing at night, you wouldn’t have that confidence to say that and then having to call up a radiology reg[istrar] on call in another hospital all takes time, doesn’t it . . . Say, at worst, half an hour.
Site D
Centres with middle tertile thrombolysis rates were keen to see the outcomes of employing machine learning included in the outputs:
How have you extrapolated your outcome data? . . . what I think would be useful to know is within that people of decision makers, who is making the decisions? . . . [I would want to see] median times with clear confidence intervals would be most useful . . . diagnosis at discharge comparative to decision-making at the time.
Site D
So, I think that is, kind of, disabilities should be part of that pathway, some type of assessment for that, for instance, the things which I do is I have to try and identify a link between the disability they get and patient kind of function – that would be helpful.
Site F
Perspectives on simulation and machine learning varied by the size and type of unit that the physicians worked in, with some participants welcoming the addition of modelling to their decision-making tool kit and others worried about the loss of their agency.
Potential routes for the implementation of machine learning feedback
Across centres, there was an understanding that modelling had the potential to identify which changes a particular centre could invest in to improve their stroke pathway:
Tell us we should do our scans quicker or hurry up with our CTAs [computerised tomography angiography] . . . Placing them on the scanner table rather than wheeling them round . . . simple things.
Site H
The SSNAP data we have is great, but it’s difficult to apply that to solutions locally. Whereas if you could apply the modelling to a local set-up and find out where the delays are consistently across a number of cases, rather than just looking at one case . . . if you do that across hundreds of cases in the same centre, then you find local solutions to increase speed.
Site H
When asked to identify the potential routes by which machine learning might inform or improve future stroke management, physicians replied with suggestions that matched particular issues with which they were grappling.
Interviewees in low-thrombolysing centres wanted a tool that could help them to improve care with a particular patient or type of patient, via a prototypical patient:
[I] would value a prototype patient . . . where it showed you which hospital would or wouldn’t thrombolyse . . . I would trust the data . . . we could get some advice on where to improve . . . there might be some big gains from that, if we did it.
Site B
If we had the information that over the country [about older frail people from care homes], it would probably give those hospitals that are more cautious, more confidence to give that thrombolysis.
Site D
People are quite afraid of the risk of bleeding and things like that. If they produce a type of individualised risk and benefit for the patient, on the information that’s provided on algorithm, that would be very helpful . . . and also the ability to be updated quickly, that would be very helpful, because the texts change every year and then sometimes you can’t keep up with all those protocols and pathways.
Site H
Physicians working in the highest-thrombolysing unit who expressed greater familiarity with SSNAP, as well as other performance indicators, wanted a more sophisticated instrument that could compare treatment across consultants or centres:
Consultant 1 (site H):
. . . internally we tend to look at consultant level data, just by looking at the thrombolysis data and picking that apart. But obviously the numbers are small, so the data can be quite varied . . . but I don’t mind seeing it at consultant level information as well as, then, hospital.
Consultant 2 (site H):
You take a prototypical patient and apply them to the algorithms that you’ve constructed for our hospital and see what pops out the other end . . . These things rarely provide an answer; they just point you to something you can reflect on . . .
Perspectives on the potential routes for the implementation of machine learning feedback were informed by physicians’ beliefs about their current needs, with the idea of a prototype patient proving popular. However, there was variance in beliefs about what variables should be included and whether its objective should be to direct patient care or to act as a quality improvement tool.
Anticipated consequences of stroke pathway feedback
Two physicians, both of whom worked on their own, in lower-thrombolysing centres, were sceptical about the consequences of changes to the stroke pathway. Having identified that they found decision-making about thrombolysis difficult, both then questioned the evidence base for increasing the rate of thrombolysis:
I do think it’s about, kind of, the personality of the person deciding it, it is very subjective, is thrombolysis, I mean, I know we have all the guidelines as to who we should and shouldn’t thrombolyse, but, you know, some consultants will aggressively continue to reduce the blood pressure with as much i.v. [intravenous] medication as they can until they can thrombolyse, others will say, well, you know, a few doses and if it doesn’t come down, OK, it’s probably not meant to be, so yeah. And you know, I think I personally just sort of very much stay within the exact rules for whether you should or shouldn’t thrombolyse.
Site B
. . . more thrombolysis doesn’t mean better care . . . when I hear of hospitals that are thrombolysing, . . . 20-odd per cent, I do sometimes question them. Are they really thrombolysing strokes? Is the clinical diagnosis of stroke really robust enough or are they thrombolysing mimics and then putting that into their SSNAP data anyway, just to make them look good? And then their mortality rates are lower because they’ve given thrombolysis non-strokes anyway. So, some of me is – I’m a bit cynical with, of the SSNAP data sometimes, from some of the sites that appear to be doing really well.
Site E
In contrast, interviewees those in higher-thrombolysing centres had a more balanced perspective on the perceived benefits of implementation of machine learning in the stroke pathway, but identified the likely enduring challenges of thrombolysis decision-making:
Consultant 1 (site H):
There’s been a gazillion studies looking at how to give tPA [tissue plasminogen activator] quicker, so the question, I think, for you guys, is what’s going to be different about this, compared to everybody else’s that tells us to get ready, get the ambulance there quicker, be more streamlined, have a checklist, der, der, der, you know, what’s going to be different?
Consultant 2 (site H):
Outcomes data with a comparator is a disaster, what does it mean? . . . I think you’re going to end up with a league table, but basically we already have one with SSNAP.
Consultant 1 (site H):
The implementation of artificial intelligence and automated reporting of scans would change the picture, would change the landscape, let’s say, of the speed of thrombolysis.
Consultant 2 (site H):
To be fair, the only aspect of machine learning I can see in this is the thrombolysis decision-making process. The rest is all straightforward factors . . . The only two parts machine learning is going to help is if the machine can actually interpret the head scan for us, which is really part of the decision to treat or not treat, and that’s the only real machine learning aspect of this, the rest is not . . . your decision to treat or not treat . . . That’s the difficult part. That’s the grey area where everyone does a different thing.
Participants were clearly curious about machine learning, and they welcomed the opportunity to discuss its potential benefits for stroke pathway feedback. Findings suggest that stroke physicians have doubts and concerns about the ability of machine learning to improve the pathway, suggesting that further dialogue is required.
