This chapter summarises the key study challenges and lessons learned. These are relevant to many primary care studies, but particularly those involving large numbers of sites and those intending to use (or develop for use) electronic trial platforms. The lessons learned are presented as ‘recommendations for future practice’ and grouped by those responsible for the activities of site identification, site set-up, site training, platform development, platform installation, troubleshooting, platform function monitoring and data management. Finally, there are two recommendations for national stakeholders, including the Department of Health and Social Care (DHSC) and NIHR.
Clinical Research Network co-ordination of site recruitment
From a study perspective, CRN facilitation of site set-up was frustrating. The CRNs were reluctant to tell the study team which practices had been invited and how many times, and there was little feedback regarding why practices did not wish to participate. The study team was clear that there was no prerequisite for sites to be ‘research active’, as the TRANSFoRm platform software was intended to guide even novice recruiters through recruitment. Indeed, for some studies, recruiting from research-naive sites could be important to ensure the generalisability of the final study sample.79 In this case, the team sensed that the CRNs were approaching research-active sites only.
Recommendation
- 1.
Clinical Research Networks (CRNs) should keep logs of which sites have been invited, when and how many times. These should be shared with study teams to populate CONSORT flow diagrams and allow a description of the generalisability of the recruiting sites.
Site set-up
We worked closely with the sponsor to use risk-adapted site set-up and study conduct approaches, aiming to reduce the burden on general practices. Despite this, there were still long delays in securing the required documentation from the general practices, particularly the delegation logs and curricula vitae, for which the sponsor required wet ink signatures. Despite clear instructions, these were regularly incomplete or incorrect, with signatures missing. As staff were inevitably busy, having to return a document because of a small mistake often resulted in several days/weeks of delays, and the paperwork was often forgotten and needed to be chased. This added to the workload of the study team, who were often chasing multiple sites for various documents/amendments to documents, and prevented the site from receiving the green light from the sponsor.
The study team kept a spreadsheet record of all documents received from each site within each of the 15 CRNs. A portable document format (PDF)/Microsoft Word (Microsoft Corporation) copy of each document was stored electronically in dedicated site folders. Although this system worked to a degree, a study of this size may have been better employing a more robust electronic document management system that could highlight missing or wrong documentation. This would have made it easier for the study team to chase sites.
Recommendations
- 2.
Sponsors should consider accepting electronic versions of delegation logs with electronic signatures. These should be designed so that submission of incomplete logs/curricula vitae is not possible.
- 3.
For large distributed trials with many sites, a robust electronic data management system to track documentation should be employed.
Site training
Because of the number of sites expected to be needed for REST, it was decided that online training (see Appendix 1) would be a more appropriate and efficient way of carrying out study training. It was predicted that busy GPs and nurses would complete the training at times that were convenient to them, and that this would expedite opening to recruitment. Unfortunately, this did not prove to be as an efficient method as hoped because clinician time is not ring-fenced for remote training as it is for face-to-face training, and staff sometimes took weeks or months to complete online training. Without the face-to-face engagement, it was difficult to encourage site staff (some of whom the study team had not worked with previously) to complete the training in a timely manner. With more studies employing remote training mechanisms, research-active general practices need to be encouraged to allow time for staff to complete remote training.
Recommendation
- 4.
When online site training is used, studies should provide training using a website that provides automated reminders and notifies the sponsor and study team when training is complete.
Electronic platform
Development
The TRANSFoRm platform was a novel electronic platform that was considered necessary for the REST trial as most general practices would see only a few cases of AOMd each season. The vision was that the system would ‘take the hand of the clinician’ and guide them quickly through eligibility, randomisation, baseline data collection and randomisation, all of which would be integrated within the EHR to autopopulate relevant data, thereby saving time, minimising data entry and minimising data entry errors. The platform was considered key to facilitating recruitment and was designed to support many sites, reducing the burden on the trial team and increasing the quality of the data.
In the event, we seriously underestimated the variety of computer configurations in English general practices, the complexity of IT support arrangements, the difficulties of integrating and ensuring smooth platform function, and the resources required to overcome these obstacles. These led to significant delays to the first general practice opening to recruitment, delays in recruiting further general practices, delays in site set-up and, ultimately, reduced participant recruitment.
The failure of the TRANSFoRm platform to provide data regarding which sites were and were not managing to recruit meant that the study team was virtually blind to, and unable to support, site recruitment activity. We have minimal information regarding the frequency of pop-up triggers (see Table 3) and know of only 30 children being invited to participate (see ). Compared with our recruitment assumptions (see Table 1), 30 invitations would have resulted in two or three children recruited. We are aware of instances of potentially eligible children not being recruited because the TRANSFoRm platform was not available. Practice staff found this frustrating, with some sites withdrawing from the study.
Detailed platform specifications were drafted (see Detailed TRANSFoRm technical specification for REST v1.4; available from study authors on request). Some were specific to REST, but some could be applied to other studies. For example, some were intended to provide estimations of the generalisability of the final trial sample in relation to the characteristics of potentially eligible children invited but declining. Some of the TRANSFoRm platform’s software components had incompatibilities with certain versions of the Windows operating system, and this made automated updating of the TRANSFoRm platform’s DNC difficult. Other software, such as EHR software, would be updated without prior warning, with updates proving incompatible with the TRANSFoRm platform’s software.
Recommendations
- 5.
Electronic trial platforms should be used to harness the unprecedented opportunities to monitor, measure and test recruitment assumptions, identifying where in the recruitment process, from presentation to consent, the key ‘drop-offs’ occur.
- 6.
All necessary platform preparatory activities and required resources should be clearly defined, taking care not to underestimate.
- 7.
The skills needed to set up a trial platform and to set up a trial are distinct and complementary. Ideally, teams should be co-located to ensure that platform specifications meet individual trial requirements.
- 8.
Platform software needs to be compatible with all practice software systems.
- 9.
Closer integration with EHR providers would prevent incompatible updates (note that this could be obviated if national criteria were agreed or if the trial platform was integral to the EHR).
Installation
Owing to the lack of a standardised set of security requirements for software in general practices, some providers requested further assurances from NHS Digital (Leeds, UK) and for additional security features to be added to the system. We engaged with NHS Digital and NHSX (Leeds/London, UK), and they confirmed that they consider our software to be safe for installation, but declared that any detailed technical audit was outside their remit and was the responsibility of EHR providers. This required us to negotiate approval with the EHR providers, The Phoenix Partnership (TPP; Leeds, UK) and Egton Medical Information Systems (EMIS) Web (EMIS Health, Leeds, UK).
Although IT support is formally handled by the CCGs, it is often outsourced to independent third parties. The installation of software such as the TRANSFoRm platform is typically not covered by the contracts governing these outsourced relationships, and the criteria for establishing the safety of new software prior to practice installation were not clear. Some third parties quoted prohibitively high costs to support installation. These issues came to light only after the practices were recruited. The REST team worked hard to obtain CCG approvals from many areas, but some approvals took months, with key areas such as the Nottinghamshire Health Informatics Service (NHIS) Change Advisory Board (CAB) and the Devon CCG arriving too late.
Recommendation
- 10.
Project teams need to work closely with EHR providers and CCGs from the study outset to agree on the software deployment process and the validation criteria required (note that this could be obviated if national criteria were agreed).
In addition to CCG approval, installation agreement was part of the site agreement, meaning that national- and CCG-level approvals were required before practices could sign the recruitment contract, which included identifying the site personnel involved, machines and training.
Recommendations
- 11.
A pilot installation incapable of being used for recruitment (and, therefore, not a site agreement requirement) should be performed on one computer in each practice, tested, and left to run for a week, before the software is installed on other machines.
- 12.
Where software reinstallation is required, it must be undertaken in a way that does not disrupt the work of the practice.
Troubleshooting
To try to assist the under-resourced TRANSFoRm team (based in London), some troubleshooting tasks were reassigned to the RCT team (based in Bristol). However, this was less effective than was hoped because the Bristol team did not have the IT skills and experience to efficiently address the challenges, leading to further delays.
Recommendations
- 13.
Electronic study platforms require teams dedicated to (1) development and (2) troubleshooting.
- 14.
Careful consideration should be given to who is responsible for troubleshooting. Although it may seem obvious that this would be performed by the trial team (as it involves interacting with sites), it requires awareness of platform function and, therefore, may be better provided by the platform development team.
Monitoring function
No one in the team anticipated the importance of a dashboard reporting real-time platform functionality. This can be likened to the real-time status reporting of the London Underground tube system.72 Much of the time, neither the TRANSFoRm team nor the trial team were aware that the platform had stopped working, either across all sites or at specific sites. Even now, we cannot estimate the proportion of time during which, or the proportion of practices at which, a fully functioning platform was able to facilitate recruitment. Functionality could be threatened for a number of reasons, including problems with and updates for the TRANSFoRm platform, the host EHR system, the Windows operating system or practice network configurations.
Recommendation
- 15.
Electronic trial platforms would be best served by a dashboard function to monitor and log platform functionality in real-time, providing real-time alerts and diagnostics for reduced function, and logging functionality across time and space.
Data management
At the project outset, no agreement was made on the format and types of data to be sent from the study database to the trial team.
Recommendation
- 16.
The format of the final data set to be extracted from the study database should be prespecified to ensure appropriate data format and avoid the submission of linked clinical and personal data.
National stakeholders, including the Department of Health and Social Care and the National Institute for Health Research
Many of the above challenges and solutions may be better addressed on a national level by a single co-ordinated management and implementation group. This would be particularly helpful for the NHS IT-related challenges, as NHS IT will continue adapting to meet future NHS needs and, unless research is considered as part of these changes, future software study platform developers could find it similarly difficult to ‘bolt on and hold on’ software to these changes.
We consider that the key national stakeholders should include the DHSC; NHS Digital; NHSX; NIHR; senior researchers with trial and observational study experience; EHR providers, such as EMIS Web and TPP; the Medicines and Healthcare products Regulatory Agency; Clinical Practice Research Datalink; and GP IT Futures framework.80
The prize is rich and the opportunity clear: to lead the world in the delivery of pragmatic research that quickly and efficiently develops highly generalisable new knowledge to improve patient care. The onset of the SARS-CoV-2 pandemic perfectly illustrated the need for and value of infrastructure ready to quickly respond to the changing health needs of the UK population.
In our view, the NIHR and other funders need to be part of the solution, but cannot solve it alone. We are delighted that the NIHR HTA accepted our argument that REST needed to be underpinned by an electronic research platform, but in doing so it took a risk to support the time needed to develop the platform. We are deeply regretful that we were not able to fulfil this ambition.
Recommendations
- 17.
Research funders need to formally recognise the potential of electronic study platforms if they wish to put the NHS on the leading edge of pragmatic research globally, allowing the delivery of new, near-real-time generalisable knowledge. This could also provide unprecedented opportunities to monitor, measure and test recruitment assumptions, identifying where in the recruitment process, from presentation to consent, the key ‘drop-offs’ that influence final study sample representativeness occur.
- 18.
NIHR and research funders should consider convening a meeting of national stakeholders to define a strategy for the development, implementation and ongoing management of electronic study platform software.
Trial management
The TMG and co-chief investigators were aware throughout the study of the delays to the delivery of the TRANSFoRm platform, despite repeated reassurances that the platform was almost ready. The co-chief investigators found it challenging to manage these delays, particularly given the academic nature of the collaboration with the partners responsible for platform delivery, and the need to maintain good working relationships. Although we feel that good relationships were maintained, we reflected with our host (Bristol, North Somerset and South Gloucestershire CCG) and the sponsor (University of Bristol) on how to manage delays in future studies. This meeting resulted in the host proactively communicating their role of enforcing contractual obligations to all their hosted studies, which includes engagement with project managers between management meetings and the creation of the CCG Contractor Escalation Guidance document (see CCG Contractor Escalation Policy; available on request from study authors), which explains and guides the assistance that the host can provide to chief investigators and project managers with underperforming collaborators.
