Introduction
This chapter considers how the impact of a randomised trial might be measured in terms of its contribution to a subsequent systematic review. After an introduction, the results of the literature search are presented. The results of a search of The Cochrane Library for systematic reviews that include randomised trials funded by the HTA programme are then presented and the implications of these findings are discussed.
The HTA programme aims to fund research on technologies that matter to the NHS. In doing so, it funds mainly systematic reviews and randomised trials. Given the widely used hierarchy of evidence, which privileges randomised trials, most systematic reviews are composed of only randomised trials. Meta-analysis provides a synthesised estimate of the effectiveness of the technology. Results from systematic reviews are seen as more robust than those from particular trials and are the preferred basis of decisions by NICE and similar bodies. The two relate in that new trials are rarely funded without a prior systematic review about what current evidence exists. Many of these identify weaknesses in the body of research and recommend new research, usually randomised trials.
Systematic reviews can also provide a way of retrospectively assessing the impact that such trials made to the evidence on the clinical effectiveness and cost-effectiveness of the technology involved. Systematic reviews and their linked meta-analyses provide quantitative measures of the quality and impact of individual trials. Trials are assessed for quality to be included. The included trials indicate not only how many have reported but also the contribution (weight) of each. The resulting synthesised effect size may be statistically significant even though none of the contributing trials was, or, despite some trials showing statistically significant differences, the synthesised effect may not be statistically significant.
Although the HTA programme uses systematic reviews to identify topics for randomised trials, it does not review the impact that those trials have on any subsequent systematic reviews. To some extent it relies on this happening automatically, as the programme partly funds the Cochrane Collaboration, which has pioneered impartial, thorough systematic reviews that are in the public domain. However, the HTA programme keeps no record on the extent to which relevant systematic reviews include trials it has funded (whether new or updates). Updating systematic reviews varies widely, reflecting developments in science. Some HTA journal reports include updated systematic reviews by the triallists and others take a long time to be updated. Some updates adopt slightly different, narrower foci. No guarantee exists that a trial funded by the programme in response to an earlier systematic review will be included in that review’s update, if and when it is updated.
The HTA programme’s early adoption of the process whereby systematic reviews preceded new trials implied a cycle in which new RCTs would contribute to the updated meta-analysis so that uncertainty was reduced if not eliminated. Although methods of measuring such progress were not available when the programme started, this chapter explores if that has remained the case.
Many of the early projects commissioned by HTA were systematic reviews, all recommending further research (usually RCTs), some of which the programme went on to commission and fund. Of 109 projects that included a randomised trial to March 2011, 62 (57%) were recorded as quoting a prior systematic review in its monograph report.228 Most [109/125 (87%)] of the RCTs funded by the HTA programme were superiority, not equivalence (non-inferiority), trials. Superiority trials also made up the bulk of systematic reviews.
Owing to the time lag between identification of topic and publication in HTA Journal, almost all trials published to 2011 had been commissioned pre-2006 when researcher-initiated bids began to be funded. Therefore, almost all were on topics identified and prioritised by the programme. For RCTs that are ‘commissioned’ (i.e. defined) by HTA, the research question, the existing evidence and the need for a trial are specified.
To explore the scope for measuring the impact of randomised trials through their contribution to systematic reviews, one needs to know what the literature offers in terms of methods and the extent to which systematic reviews that include trials funded by the HTA programme exist.
Results
The results of the literature search were divided into descriptive studies, those that explored the use of systematic reviews in designing trials, those regarding VOI [and its York/Patient-Centered Outcomes Research Institute (PCORI) variant] and those assessing systematic reviews in relation to their use in clinical guidelines.
Descriptive studies
Gouldie et al.229 showed that a minority of trials published in leading journals cited previous RCTs (6/27) or systematic reviews (10/27) in determining their target sample size.
Jones et al.,230 in a review of randomised trials funded by the HTA programme between 2006 and 2008, found that 77% (37/48) cited a previous systematic review. A follow-on of this study231 found that almost all of those that lacked a prior systematic review did so for good reasons. Some were new topics (first in class), in which case a formal systematic review might not be necessary. In other instances, the previous trials were judged to be of low quality.
Use of systematic reviews in designing new trials: literature
Sutton et al.232 employed simulation methods to estimate the trial sample size linked to probability of conclusive result in meta-analyses. Their results favoured several small RCTs, rather than a single large conclusive trial.
Roloff et al.233 used a random-effect meta-analysis to estimate conditional power, which is needed to detect a specified mean effect size for a given meta-analytic result. This work showed that, in the presence of heterogeneity, no trial could be large enough to be conclusive. Instead, several separate trials would be needed.
Heterogeneity was defined as relating to genuine differences underlying the results of RCTs (heterogeneity) as opposed to variation as a result entirely of chance (homogeneity). The degree of heterogeneity has become widely used in Cochrane reviews.
Value of information literature
Value of information, a concept from decision analysis, relates to the extent of uncertainty and the types of information that would reduce that uncertainty relative to perfect information. Wikipedia provides an entertaining definition: ‘Value of Information is sometimes distinguished into value of perfect information, also called value of clairvoyance, and value of imperfect information. They are closely related to the widely known expected value of perfect information and expected value of sample information’. More formal accounts of VOI have recently been puplished.234–236
Cost-effectiveness acceptability curves, an element of the VOI approach, have been used in almost all the economic analyses in HTA-funded trials. They have also been adopted by NICE in its technology appraisal programme.
The HTA programme has made use of the VOI approach. It decided not to fund a trial based on a form of VOI from the preliminary assessment of technology for health services model.237 It funded a review of the ‘role of modelling in planning trials’,238 which favoured VOI. It also reported the results of a pilot study of the use of VOI by the programme.239 This was successful according to the authors, a University of York team that carried out the relevant analyses. A more recent review240 found that 25 of approximately 500 studies (≈5%) published by the programme had used VOI method(s).
Overall, VOI appears more useful in assessing the gaps in research than in assessing the impact of RCTs. Projects that are developed on the basis of VOI should produce less uncertain estimates of the cost-effectiveness of particular technologies. The extent to which this results in greater impact, however measured, is unknown. Research on this forms one of PCORI recommendations below.
Patient-Centered Outcomes Research Institute
The PCORI was authorised by the US Patient Protection and Affordable Care Act of 2010241 to conduct research to provide the best available evidence to help patients and those who care for them to make better-informed health-care decisions.
The PCORI was explicitly prohibited from using QALYs and cost-effectiveness in determining research priorities.242 Instead of using VOI, which implies use of cost per QALY, it developed (along with Claxton et al.234 from the University of York) an amended version that relies on cumulative meta-analysis and minimum clinical difference. This, combined with incidence/prevalence, can be used to assess the worth of funding new research. It involves asking if the expected net health benefits of additional evidence are sufficient to regard a particular research proposal as worth prioritising.
The scale of uncertainty is assessed from systematic reviews and meta-analysis, combined with baseline risk and incidence. These are used to express the expected consequences of uncertainty in terms of health outcomes. The approach also assumes degree of implementation.
The York/PCORI package was offered as a software package to the HTA programme in 2013, which has been exploring its use. No formal evaluation appears to be taking place.
This approach shares the VOI emphasis on prospectively prioritising elements of research. Whether or not reduced uncertainty leads to greater impact remains unknown.
Literature on systematic reviews and clinical guidelines
A different approach for valuing trials considers the scope for basing clinical guidelines on systematic reviews. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) group provides tools to assess the quality of information in relation to its use in guidelines. GRADE ‘is a well-developed formal process to rate the quality of scientific evidence in systematic reviews and to develop recommendations in guidelines that are as evidence-based as possible’.243
The GRADE tool is used by NICE along with many similar organisations elsewhere. Given the aim of the HTA programme in relation to meeting the needs of the NHS with high-quality evidence, GRADE provides a tool for assessing the extent the usefulness of that evidence for clinical guidelines.
The GRADE tool recommends that:
The strength of recommendations should be expressed using two categories (weak/conditional and strong) for or against a management option and the definitions for each category should be consistent with those used by the GRADE Working Group.
Decisions about the strength of the recommendations should be transparently reported.
However, an evaluation of GRADE suggests that it has had little impact on which systematic reviews are included in clinical guidelines.244
Results of search of The Cochrane Library for systematic reviews that included trials funded by the Health Technology Assessment programme
The Cochrane Library was searched to explore the extent to which systematic reviews that included randomised trials funded by the HTA programme existed. This section provides preliminary results. As shown in , 28 (around one-quarter) out of the 121 eligible trials (excluding four feasibility or pilot trials) published up to 2011 featured in a subsequent Cochrane systematic review. This indicates that subsequent systematic reviews cannot be assumed.
Trials published by the HTA programme to 2011 that featured in subsequent Cochrane systematic reviews
Caveats relate to the limited nature of the search and its restriction of the search to The Cochrane Library. Although a more thorough search might reveal a few more instances, we are confident the results presented here indicate the limited extent to which trials funded by the HTA programme feature in systematic reviews.
Only around half of the meta-analyses in these systematic reviews indicated that the intervention was superior, in that the 95% confidence interval for the primary outcome excluded no difference.300 For the rest of the comparisons, the difference was not statistically significant. Non-superiority cannot be interpreted statistically as proof of no difference (whether non-inferiority or equivalence). However, although some authors claimed non-inferiority, this was not strictly justified. The policy implications resulting from systematic reviews that do not demonstrate superiority deserve separate attention, if only because such results are so common.
Several other points are worth noting from . First, the weight of the HTA trials in the meta-analysis varied widely. For some comparisons in eight reviews,252,254,258,264,269,273,283,297 the trial funded by the HTA programme had a weight of 100%, indicating that it provided the only evidence for that particular comparison. For the others in ,252,254,258,264,268,272,282,296 the weight was much lower, indicating a lower contribution of that trial to the synthesised result. This is partly because of other trials reporting after the HTA trial was funded.
Second, the HTA-funded trials tended to have higher weights for comparisons relating to patient-related outcomes and/or longer time frames. This indicates that the contribution of a trial to a systematic review can take many forms, some of which may be more relevant to those commissioning health services than others.
Given that only around one-quarter of HTA-funded trials are included in a subsequent systematic review, more work is required on their contribution to knowledge.
Discussion
The chapter has shown that the HTA programme contributes only one side of the loop linking systematic reviews and trials. Although it usually carries out a systematic review before funding a trial, it does not follow up with any analysis of the contribution that trial makes to any subsequent systematic review.
The York/PCORI approach used in 2015 by the HTA programme for deciding when to fund trials is important and deserves to be evaluated. However, without subsequent analysis of the trials funded, this will not help with assessing the contribution any trial funded makes to the body of evidence.
The GRADE tool offers a way to assess how complete the evidence in systematic reviews is for the purpose of drawing up clinical guidelines. This may offer a way for the HTA programme to assess the extent to which a particular trial has contributed and if another is required.
The finding that only around one-quarter of HTA-funded trials are included in a subsequent systematic review shows that a contribution by a trial to systematic reviews cannot be assumed. Furthermore, the fact that around half of the Cochrane systematic reviews to which a trial contributes do not demonstrate superiority prompts questions as to how to assess the value of these.
Research recommendations
There is merit in using existing systematic reviews to assess the impact of trials funded by the programme. When systematic reviews of important topics are lacking, the programme should consider funding them.
When the topic of a trial may become part of a clinical guideline, GRADE should be employed to assess the completeness of the evidence. The exploration by the HTA programme of the PCORI/York approach of assessing the need for a trial in relation to existing systematic reviews should be evaluated.
Research on the extent to which projects and programmes that employ either VOI or the PCORI approach achieve greater impact would be valuable. As the HTA programme has made some use of both VOI and PCORI, the relevant projects might be considered for case studies as well as for bibliometric analysis.
Work is also required on the policy implications of non-superiority results in systematic reviews of superiority trials.
