Essay 2Randomised controlled trials of complex interventions and large-scale transformation of services

Cluster randomised controlled trials

Cluster randomised controlled trials (cRCTs) have emerged partly as a means of addressing such problems. The units randomised are pre-existing, natural or self-selected clusters. Cluster members all have an identifiable feature in common (such as patients in a single general practice), and outcomes are measured in all or in a representative sample. The clusters can vary widely depending on the setting, and range in size from families to entire communities.

Methodological discussion of the cRCT appears to have begun in the field of education in the 1940s,⁸ but the design saw only sparse use before the 1980s. However, the last half-century has seen a steady increase in cRCTs published in the medical literature: from one per year in the 1960s to over 120 in 2008.⁸ The main implication of the cRCT design is that the outcomes of individuals within the same cluster tend to be correlated, which must be accounted for in the analysis. The statistical measure of correlation is known as the intracluster correlation coefficient (ICC), defined as the proportion of variation in the outcome that can be explained by the variation between clusters. Sample size estimates need to account for this, taking into account both the size of the clusters and the ICC. Reliable estimates of ICCs are essential for robust sample size calculations. There is increased understanding of the factors that affect clustering and therefore the likely magnitude of an ICC. For example, Campbell et al.⁹ demonstrated that ICCs are sensitive to a number of trial-related factors, particularly the setting and the type of outcome. Analysing a range of data sets, mainly from the UK, they found that ICCs were significantly higher for process than outcome variables and for outcomes in specialist settings compared with primary care. The effects of disease prevalence and trial size were less clear-cut.⁹ Other studies, meanwhile, suggest that ICC is associated with prevalence for binary outcome measures.¹⁰

There is also growing understanding of how to calculate sample sizes for cRCTs, including scenarios in which the size of individual clusters varies.¹¹ This has been incorporated into standard statistical packages, and relevant online tools are also available.¹²

The third development has been the publication of reporting guidelines for cRCTs. These guidelines highlight that it is important to indicate the level at which the interventions were targeted, hypotheses were generated, randomisation was done and outcomes were measured.¹³ In 2012, specific guidance was published, based on the 2010 version of the Consolidated Standards of Reporting Trials (CONSORT) statement and the 2008 CONSORT statement for the reporting of abstracts.¹⁴ The guidance includes a checklist of items that authors should address when reporting cRCTs, in conjunction with the main CONSORT guidelines.¹⁵

There has also been the development of an appropriate ethical framework, The Ottawa Statement on the Ethical Design and Conduct of Cluster Randomised Trials. This sets out 15 recommendations, providing guidance on the justification for a cRCT; the need for ethical review; the identification of research participants; obtaining informed consent; the assessment of benefits and harms; and the protection of vulnerable participants.¹⁶

Stepped-wedge trials

The stepped-wedge trial has emerged as an alternative to the standard parallel-group cRCT and is increasing in popularity for evaluating complex interventions and large-scale transformations.¹⁷ The design includes an initial period during which no clusters are exposed to the intervention and therefore act as controls. Subsequently, at regular intervals (the ‘steps’), one cluster (or a group of clusters) is randomised to cross over from control to intervention. This process continues until all clusters have crossed over and are exposed to the intervention (Figure 2.1). The appeal of the design is that, at the end of the study, there will be a period when all clusters are exposed, but the order of intervention delivery is determined at random. Data collection continues throughout the study, so that each cluster contributes observations during both control and intervention observation periods.¹⁹

FIGURE 2.1

Stepped-wedge trial schema. Shaded cells represent intervention periods, blank cells represent control periods and each cell represents a data collection point. Reproduced from Brown and Lilford under the terms of the Creative Commons Attribution Licence (more...)

The rationale for using this approach, as well as specific methodological considerations, has been described in more detail elsewhere.^19,20 However, the principal advantage is that it may more adequately reconcile the differing needs and priorities of policy-makers and service managers on the one hand, and evaluators on the other. It has intuitive appeal, as each cluster eventually receives the intervention and acts as its own control. As a consequence, it is particularly suited to interventions in which there is a lack of evidence of effectiveness but a strong belief that they will do more good than harm,²¹ such as efforts to improve hand hygiene. It also promotes the phased implementation of novel interventions, permitting time for staff training, if this is required. In addition, the researcher may study the effect of time on an intervention, such as whether or not adoption or learning is maintained.

Nevertheless, because data collection usually takes place at each time point in the study, the data burden is potentially significant. It may also take time before every cluster receives the intervention. In addition, the analysis is more complex than that required for a conventional cRCT. Sample size calculations and analysis must make allowance for both the clustered design and the confounding effect of time.²⁰ In a stepped-wedge design, more clusters are exposed to the intervention towards the end of the study than in its early stages. Consequently, the effect of the intervention might be confounded by any underlying temporal trends if, for example, there has been a general move towards improved patient outcomes, external to the study.¹⁹ Of course, judging whether or not an intervention is likely to do more good than harm is complex, especially when considering wider issues such as opportunity costs. A stepped wedge may mean that all clusters are exposed to an ineffective intervention if initial estimates of effect prove unfounded.

Selecting a randomised approach

Stepped-wedge trials and cRCTs both offer a means of evaluating complex interventions and large-scale transformations. In what situation should one be preferred? Consider a hypothetical study, involving a group-based weight loss programme with internet support. Early pilot investigations suggest that the intervention is effective, with a primary outcome of weight loss after 6 months. There is interest locally in implementing the programme; eight obesity clinics have provisionally agreed to participate.

In this example, we assume a minimum duration of follow-up of 6 months, and monthly introduction of the intervention into new clusters. In the event that the intervention is effective, the final results would be available in 14 months using a stepped-wedge design. By that stage, all the study clusters would have at least 6 months’ experience of using the intervention, and have been reaping the benefits. In contrast, if a parallel-arm cRCT were used, this result would have been known within 6 months. Four of the clusters would have been delivering the programme since the start of the trial. However, it would now be necessary to ‘fast track’ the intervention in the four control clusters.

On the other hand, consider if the intervention were not effective. With a stepped-wedge design, the final results would again be available in 14 months. However, by that point, all of the clusters would have been exposed to an intervention that does not work, and all would have been delivering it to patients for at least 6 months. Consequently, ethical scrutiny of this type of trial is important. In addition, because data collection usually takes place sequentially, the researchers would have collected a full set of trial data on 14 separate occasions (although a smaller number of data collection periods could be considered). If a cRCT were used, the results would be known within 6 months. Only half of the clusters would have been exposed to an intervention that did not work, and trial data would have been collected only once.

Key questions to be addressed before commencing such a trial include:

• What is the strength of the evidence that the intervention is more likely to do more good than harm? Considerations of costs and opportunity costs may be relevant.
• As well as effectiveness, what other issues are being assessed by the trial, such as learning and possible decay of the intervention effect?
• How quickly could an effective intervention be rolled out to control clusters if a cRCT were conducted?

The optimal conditions for a stepped wedge are likely to be when there is a lack of evidence of effectiveness but a strong belief that the intervention will do more good than harm.²⁰ It may also be relevant when measurement of the effect of the intervention over time is important, including evaluation of staff learning and the effect of the service ‘bedding down’. In contrast, a cRCT may be more appropriate when the evidence of effectiveness is not clear-cut and rapid implementation in control clusters is possible. cRCTs have the added advantage that half of the study sample is ‘protected’, should the intervention not work, and the data collection burden is minimised.

In many situations, the concerns of those commissioning the evaluation may outweigh methodological issues. The constraints under which policy-makers and service managers operate may mean that a stepped wedge is the only way to gain agreement to randomisation, because all clusters will receive the intervention during the research. As a consequence, stepped-wedge designs may be particularly useful when policy dictates that an initiative will be rolled out widely, and there is scope to do this in a stepped way to enhance evaluation.

There are examples of successful cRCTs of complex policy interventions, which have successfully negotiated issues around randomisation and implementation, such as a cRCT of a scheme to provide free healthy breakfasts in Welsh primary schools.²² Of the 608 schools invited to participate, 111 agreed to be randomised and participate in data collection activities. Stepped-wedge designs have the potential to appear more attractive to stakeholders and may encourage higher levels of participation.

Pragmatic and explanatory attitudes to trial design

The PRagmatic-Explanatory Continuum Indicator Summary 2 (PRECIS-2) tool is designed to support trial design decisions consistent with the intended purpose of a planned RCT. This tool is especially useful if the intention is to design a RCT that will have results widely applicable within usual care at whatever level of the health service this may be, from primary to tertiary care settings. PRECIS-2 consists of nine domains: eligibility criteria, recruitment, setting, organisation, flexibility (delivery), flexibility (adherence), follow-up, primary outcome and primary analysis (Figure 2.2). These are discussed until a consensus is reached that that particular element of the trial is designed in a way which promotes the goal of the trial, be it a real-world applicability or research designed purely to understand efficacy. The process makes research teams more aware of the range of opinions and can help to ensure that design decisions are matched to the needs of those who will ultimately use the results of the trial.²³

FIGURE 2.2

The PRECIS-2 wheel. Reproduced from The PRECIS-2 tool: designing trials that are fit for purpose, Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M, vol. 350, p. h2147, 2015, with permission from BMJ Publishing Group Ltd.

Challenges in intervention design

As the revised MRC complex interventions guidance highlights, best practice is to develop complex interventions systematically and then to test them via pilot studies, before moving on to an exploratory and then a definitive evaluation.¹ In practice, however, policy implementation often does not follow this model, leaving researchers with less opportunity to influence intervention design or delivery. An example from England would be the government’s population-wide prevention programme for cardiovascular disease (the NHS Health Check programme), which was rolled out despite a number of uncertainties about the evidence base underpinning this approach and how best to encourage the necessary uptake among patients to achieve the expected gains.³² Ideally, all intervention development should begin with identification of a relevant evidence base. This will often need to be supplemented by the application of appropriate theory, which may lead to greater probability of an effective intervention than a purely empirical or pragmatic approach.³³

Even in cases where researchers cannot influence the design and implementation of an intervention, they can usefully help decision-makers to develop a theoretical understanding of the likely process of change by articulating the rationale behind it. This includes the changes that might be expected, and how change is to be achieved. A ‘logic model’ can help clarify causal assumptions, which may be drawn from behavioural, social science or organisational theory. Making explicit the causal assumptions and mechanisms of effect can allow external scrutiny of its plausibility and help evaluators decide which aspects of the intervention or its context to prioritise for investigation.³

Increasing attention is being paid to the possibility of researchers and decision-makers working together to design and subsequently evaluate interventions. One such example is the Practical Approach to Lung Health in South Africa (PALSA) trial, conducted in South Africa.³⁴ From the start, researchers worked with front-line clinicians and managers to develop a guideline for the management of respiratory disease, which was tested and refined prior to implementation and evaluation. This approach is not without controversy, not least because decision-makers typically have a commitment to making an intervention work, which may impact on the evaluation and interpretation. For example, a metaregression describing features of effective computerised clinical decision support systems demonstrated that studies conducted by the system developers were more likely to show benefit than those conducted by a third party.³⁵

Importance of process evaluation

Randomised controlled trials remain the best method for making causal inference and providing a reliable basis for decision-making. However, the basic RCT design cannot assess how or why an intervention does or does not achieve outcomes. To understand this, process evaluations should be routinely included in all evaluations of complex interventions to explain discrepancies between expected and observed outcomes, to understand how context influences outcomes and to provide insights to aid further implementation.¹

Process evaluation can usefully investigate how the intervention was delivered, providing decision-makers with information about how it might be replicated. Issues considered may include training and support, communication and management structures, and how these structures interact with attitudes and circumstances of staff to shape the intervention. Process evaluations also commonly investigate the reach of interventions, for example whether or not the intended audience comes into contact with the intervention, and how.³ On the other hand, interventions may work, but in ways that were not expected or predicted.³⁴ Qualitative or quantitative process evaluation may allow modification or updating of ‘logic models’ to account for the new data and plan further research. Further consideration is given to the role of qualitative methods in evaluative research in Essay 7.

The recently published MRC guidance on process evaluation of complex interventions provides a general framework for conducting and reporting this type of study.³ Grant et al.³⁶ also provide a framework for design and reporting which is specific to cRCTs and encourages researchers to consider both the processes involving individuals in the target population on whom outcome data are collected and the processes relating to the clusters within which individuals are nested.

‘Realist’ evaluation methods purport to go beyond conventional evaluations to consider not just ‘what works’, but ‘what works, for whom, and in what circumstances?’ In order to answer these questions, realist evaluators aim to develop and continually update relevant programme theory, to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context.

Although realist methods are often contrasted with conventional RCTs and systematic reviews, the differences may not be so acute.² For example, quantitative methods linked to RCTs exist for the analysis of mediation (to explore ‘how’) and moderation (to assess ‘for whom’), which can usefully complement qualitative work.³⁷ Bonell et al.² also propose that ‘realist’ aims can be examined in the context of RCTs. Such ‘realist’ RCTs should aim to examine the effects of intervention components separately and in combination, for example using multiarm studies and factorial designs; explore mechanisms of change, for example how pathway variables mediate intervention effects; use multiple RCTs to test how intervention effects vary with context; draw on complementary qualitative and quantitative data; and be oriented towards building and validating ‘mid-level’ programme theories which would set out how interventions interact with context to produce outcomes.² ‘Realist’ RCTs, which not only provide an unbiased average effect estimate but also test and refine programme theory, are a potentially important extension of existing pragmatic RCT methods. Examples where this approach has been used successfully include a cRCT assessing the impact of school fruit ‘tuck shops’ on children’s consumption of fruit and snacks in deprived areas in south Wales and South West England³⁸ and an evaluation of a peer-led intervention that aimed to prevent smoking uptake in secondary schools in England and Wales.³⁹

The addition of process evaluations to RCTs does highlight potential tensions between quality and depth of evaluation, and its speed and efficiency. Process evaluations may require detailed qualitative and observational work, which may add significantly to the costs of the research and its duration. There is also a tension between an ‘independent’ process evaluation designed to make sense of the eventual trial results, and the desire for feedback from the evaluation mid-stream to deal with issues that may be limiting effectiveness.³

Designing for implementation

To successfully improve health care and population health, we need complex interventions with a degree of flexibility, resilient to contextual variation and therefore more likely to be implemented widely. We also require a clear understanding of contextual dependencies (such as target group, resources, system requirements) and system impacts. As we have outlined, attention to implementation issues in intervention design and understanding how or why an intervention works in the trial setting are both crucial to determining whether or not the intervention can be reliably disseminated to other settings.

The traditional linear ‘phases of research’ model remains influential. However, this may not be optimal for complex interventions, because it assumes that real-world effectiveness research naturally follows from successful efficacy research. As we outlined at the start of this essay, complex interventions interact with the context in which they are delivered. Contextual factors that may moderate the effectiveness of the intervention across settings and populations may not be adequately considered in design, which means that an intervention proven to be efficacious in one context may fail to demonstrate effectiveness when implemented in contexts for which it has not been designed.

Work in implementation science seeks to challenge the linear model of translation. It has been suggested that the characteristics which cause an intervention to be successful in efficacy research (e.g. intensive, complex, highly standardised) are fundamentally different from, and often at odds with, programmes that succeed in population-based effectiveness settings (e.g. having broad appeal, being adaptable for both participants and intervention agents).⁴⁰ The Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) model proposed for evaluating complex interventions is intended to refocus priorities on wider issues, and give balanced emphasis to internal and external validity.⁴¹ The model seeks to determine the characteristics of interventions that can reach large numbers of people, especially those who can most benefit; be widely adopted by different settings; be consistently implemented by staff with moderate levels of training and expertise; and produce replicable and long-lasting effects (and minimal negative impacts) at reasonable cost. An alternative model, normalisation process theory, highlights a series of constructs representing different types of work that people do in implementation that could inform intervention design, and linking it to ideas about agency, social and cognitive mechanisms, and collective action from sociology and psychology.⁴² A range of other relevant frameworks and models have been reviewed.^43,44 For further discussion of current issues in implementation, see Essay 8 in this volume.

Greater attention needs to be paid to planning and documenting intervention reach, adoption, implementation and maintenance. One solution is to extend feasibility and pilot studies. Such studies would not only optimise evaluation methods, and further develop underlying theories or logic models, but also allow detailed consideration of how interventions would be adopted, implemented and maintained in real-world contexts. However, such methods may increase the cost and duration of evaluation and may not always be feasible.

Trial speed and efficiency

As noted in Selecting a randomised approach, a key limitation of current RCTs is their lack of speed and high cost, and there is significant interest in ways of making RCT delivery more efficient.

Randomised controlled trials rely on patients, staff and organisations to volunteer to take part. However, patient recruitment and retention remains a significant problem⁴⁵ and the evidence base for recruitment and retention is very sparse.^46,47 Many health organisations are disengaged with the research production infrastructure, which has implications for the generalisability of findings. To maximise replicability, studies need to be conducted in a wide range of settings to avoid the clear gaps between the populations in research and those to which the results are expected to apply.

There is an urgent need for research to support the more efficient delivery of RCTs.⁴⁸ This might involve use of qualitative research to explore delivery⁴⁹ or patient and public involvement to design trials more sensitive to the needs of patients and other stakeholders.⁵⁰ More consistent outcome measurement and reporting⁵¹ might avoid excessive measurement burden. Methodological studies may be usefully embedded in existing trials to grow the evidence-base for efficient delivery.^52,53 The cohort multiple RCT uses routine measurement of patients in a cohort to facilitate both recruitment and outcome measurement. Routine measurement of patients in the cohort eases the identification of patients for RCTs, while the cohort may provide a recruitment process which more naturally replicates clinical practice.

There are exemplars for more efficient trial delivery. The Ontario Printed Education Message Trials demonstrate the possibility of conducting large-scale, randomised studies at relatively low cost.⁵⁴ General practitioners in Ontario, Canada, were randomised by practice to receive a range of reminders about retinal screening in diabetic patients. The reminders were mailed to each physician in conjunction with a widely read professional newsletter. All general practitioners in the province were included in the study, and hence the result provides both a pragmatic and a reliable assessment of the intervention in that context. Alongside a study of the effectiveness of the various interventions,⁵⁴ the authors also conducted a theory-based process evaluation to assess physicians’ intentions related to referring patients for retinopathy screening, as well as their attitude, subjective norm and perceived behavioural control.⁵⁵

Medical care and associated research is undergoing something of a paradigm shift with the introduction and use of electronic health records. These provide major opportunities owing to their scope and amenability to analysis. The idea of a ‘learning health-care system’ is based in part on better use of these data to support rapid evaluation (including RCTs and other quasi-experiments) and system-wide learning.⁵⁶ The potential power of electronic health records in the context of RCTs is beginning to receive attention, particularly the potential to study large samples at relatively low cost, driving improvements in efficiency.⁵⁷ In the USA, Clinical Data Research Networks supported by PCORnet (National Patient-Centered Clinical Research Network) build capacity for conducting both RCTs and observational effectiveness studies by collating data in a compatible format from a range of different health-care providers, including integrated delivery systems, academic medical centres and safety net clinics.⁵⁸

Similarly in the UK, cRCTs have been implemented using electronic health records to evaluate the effectiveness of computer-delivered decision support tools in reducing antibiotic prescribing for respiratory tract infections in primary care⁵⁹ and improving risk factor control after first stroke.⁶⁰ Both studies made use of electronic health records available via the Clinical Practice Research Datalink (CPRD), which includes anonymised data for approximately 7–8% of UK general practices, linked to a range of other data including mortality information and cancer registrations.⁶¹ This experience provides compelling evidence that RCTs may be performed efficiently in large samples of this kind. Process evaluation may also be carried out alongside this type of study.⁶²

However, there are significant challenges. Those who contribute information to data sets such as CPRD are skilled at collecting data, and not necessarily representative of their peers. The geographical dispersal of sites in RCTs using electronic health records may present a challenge for research governance and intervention implementation.⁶³ For example, in a cRCT which used electronic records to study antibiotic prescribing, because the practices contributing to the CPRD are anonymous, permission had to be sought from 155 separate commissioning organisations for the English general practices alone. Further permissions were required in Scotland, Wales and Northern Ireland. The process took almost a year to complete before the RCT could commence.⁶³ Such requirements make it difficult to realise the potential efficiency gains of such trials.

There are always challenges concerning the accuracy of records, and these records also provide a potentially partial view of outcomes. For example, such records rarely include patient-centred outcomes such as self-reported quality of life or patient satisfaction. Sometimes, the lack of patient-oriented outcomes is less of a problem (e.g. where the focus is on professional behaviour), but there is a danger that the efficiency of RCTs using electronic health records means that patient-oriented outcomes are de-emphasised compared with ‘harder’ measures such as mortality, or policy-maker and manager priorities such as quality of care and health-care utilisation (especially hospital admissions).

Working with decision-makers

Finally, although the purpose of evaluation is to inform decision-making by policy-makers and service providers, there is currently something of a disconnect between those who commission research and those who deliver it. As we have outlined, challenges may arise because policy-makers are often keen to pilot a new intervention in favourable circumstances, and there may be an unwillingness to evaluate at all, in case the programme is not effective. Others argue that coproduction of research with decision-makers results in a loss of researcher independence. However, there are innovative examples of attempts to bridge this gap, such as the National Institute for Health Research Collaborations for Leadership in Applied Health Research and Care in England. Evaluability assessments have also been trialled in Scotland, encouraging early dialogue between researchers and decision-makers to establish whether or not and how to evaluate programmes and policies.⁶⁴

One of the principal challenges, however, is managing potentially conflicting time scales. Decision-makers typically require a swift answer, which may be challenging to deliver using rigorous methods. This may be compounded where funding and regulatory approvals must be secured before an evaluation can commence, which may make it difficult to conduct baseline assessments prior to an intervention. On the other hand, in health systems in repeated flux, managerial staff may not be in post for extended periods of time, making it challenging for researchers to establish long-term relationships with key decision-makers. In these circumstances, service needs may over-rule methodological concerns, and researchers may have to use evaluation methods that would provide the best possible answer in the time available.

Contributions of authors

Helen Barratt (Senior Clinical Research Associate, Health Services Research) and Peter Bower (Professor, Health Services Research) wrote the first draft of the essay.

Marion Campbell (Professor, Health Services Research), Laurence Moore (Professor, Public Health Research) and Merrick Zwarenstein (Professor, Family Medicine) commented on the draft and provided additional material.