Study conclusions: UTMoSt stage 1

To the best of our knowledge, UTMoSt stage 1 is one of the first studies in the world to attempt to mimic a surgical RCT using real-world data and different analytical methods to minimise confounding. A number of recent initiatives aim to prove the value of real-world data for regulatory and clinical decision-making. Funded and sponsored by the public [FDA (Silver Spring, MD, USA); US National Heart, Lung, and Blood Institute (Bethesda, MD, USA); and Harvard University (Cambridge, MA, USA)] and private institutions [Aetion Inc. (New York, NY, USA)], RCT DUPLICATE⁷³ is perhaps the best known of these initiatives. Another initiative, the LEGEND study,⁸⁶ conducted by the OHDSI collaboration, has reported preliminary results in the area of hypertension. In a multidatabase, multidrug, PS-based analysis, the authors found that well-performed pharmacoepidemiological analyses provided findings that were highly consistent with previously performed head-to-head trials (slides available for review⁸⁶). To our knowledge, all of these initiatives attempt to replicate the findings of RCTs for the study of medicines, but none has attempted to mimic surgical or medical device trials.

Our stage 1 findings show that the replication of surgical RCTs shares some challenges with studies replicating drug or medicinal product RCTs, but also has its own challenges. In common with the replication of drug RCTs, we found that even in a relatively pragmatic, post-marketing RCT, such as TOPKAT, a good proportion of actual NHS patients would not have been eligible. About 37% of patients undergoing TKR and 35% of patients undergoing UKR in the NHS would not have been eligible for TOPKAT based on their clinical characteristics. There is clearly a space for real-world data to contribute useful information on the risk–benefit and cost-effectiveness of medical devices and alternative surgical approaches and procedures for the many patients (here around one in three) for whom no RCT-derived evidence is available. UTMoSt stage 2 delivered the best available information in the absence of RCT data for NHS patients who were not eligible for TOPKAT. The results are summarised in Study conclusions: UTMoSt stage 2.

More worryingly, the generalisability of surgical and medical device evaluation RCTs can be limited by inclusion criteria that dictate which surgeons and treatment centres are eligible to participate. Surgical RCTs tend to require surgeons with a certain level of expertise with both treatments under study and seek clinicians who are in equipoise when deciding what treatment is best for the eligible patients. TOPKAT in particular used an experience-based design, including surgeons who had carried out ≥ 10 procedures (UKR or TKR) in the previous year of the same type as the allocated treatment.²⁸

Restricting UTMoSt analyses to participants operated on by surgeons with such levels of experience excluded another 9% of possible TKR participants and one in three TKR surgeons from UTMoSt stage 1. Probably because of the lower uptake of UKR, the same criterion excluded almost half of the possible UKR participants and 64% of UKR surgeons from UTMoSt stage 1. TOPKAT surgeons had completed many surgeries in their careers before TOPKAT: a median of 100 (IQR 50–200) UKR procedures and 300 (IQR 260–400) TKR procedures.⁸⁷ Although these numbers are not directly equivalent to the previous-year surgery counts that we used, they do indicate that the surgeons participating in TOPKAT probably had greater expertise than strictly required for participation. For illustrative purposes, we reported the impact of restricting the analysis to operations performed by surgeons who had performed ≥ 30 and ≥ 50 surgeries of the same type in the previous year. These criteria excluded, respectively, > 80% and > 90% of the UKR patients operated on in the NHS who were initially eligible for UTMoSt stage 1. We discuss the impact of restricting to expert surgeons below.

Our stage 1 findings demonstrated that some (but not all) of the previously approved methods for the study of drug safety and post-marketing comparative effectiveness research can also be applied to the study of implantable devices and surgical procedures. Our results showed that some PS methods could reliably approximate the TOPKAT primary outcome results (postoperative OKS). Methods that estimated the ATEs (PS stratification and IPW) more closely approximated the TOPKAT findings than other tested methods. In the full cohort analysis, only PS stratification based on the distribution of the PS in the UKR (exposed) cohort, PSS_exp, passed all of the proposed diagnostics and was classified as able to replicate the TOPKAT findings. PSS_whole and IPW came close to passing these diagnostics.

The sensitivity analysis was restricted to patients operated on by surgeons who would have been eligible for TOPKAT, based on their experience with the index surgery, resulting in findings much closer to those seen in the RCT, with ATEs of 1.32 (95% CI 0.32 to 2.33) (IPW) and 1.37 (95% CI 0.54 to 2.20) (PS stratification) compared with 1.91 (95% CI 0.20, 3.62) in TOPKAT, all in favour of UKR over TKR. PSS_whole, PSS_exp and IPW were, therefore, deemed valid and taken forward into UTMoSt stage 2.

As well as the effect of surgeon experience on the PROM OKS, we found strong evidence of an interaction between surgeon experience and the association between type of surgery (UKR vs. TKR) and 5-year revision risk. Using the three validated methods, we demonstrated that the excess risk observed among patients undergoing UKR in the whole cohort decreased dramatically when UKR was performed by surgeons with more UKR experience. UKR patients in the whole cohort had more than double the risk of 5-year revision surgery than TKR patients. However, this increased risk dropped to around 40–50%, and was no longer significantly different from TKR patients’ risk when the analysis was restricted to patients operated on by surgeons who had performed ≥ 50 surgeries of the same type in the previous year. This finding came closest to replicating the TOPKAT findings, for which the estimated odds ratio for revision was 1.40 (95% CI 0.50 to 4.00) for UKR versus TKR, but was limited by low statistical power and, therefore, needs further research in other settings.

These findings have both methodological consequences for future research in the field and clinical relevance. Given the observed interaction between the surgeons’ experience and the obtained health outcomes (both OKS and revision risk), we suggest that UKR surgery be centralised in specialised treatment centres and provided by specialist surgeons. Our results suggest that patients will have the best possible results when their UKR surgeries are performed by surgeons who perform ≥ 50 UKR surgeries per year. This is equivalent to around one UKR surgery per working week. Conversely, UKR surgeries performed by surgeons who have performed ≤ 10 UKRs in the previous year may have suboptimal patient benefit and could even lead to higher risks of revision than TKR surgery.

Study conclusions: UTMoSt stage 2

UTMoSt stage 2 provided evidence on the comparative effectiveness, safety and cost-effectiveness of UKR compared with TKR for patients with multiple comorbidities, as measured by an ASA grade of 3 or 4. These patients would not have been eligible for TOPKAT, and it is unlikely that there will be a follow-up trial to include this subpopulation. To our knowledge, the results summarised here are the best-quality data available to date for the approximately 15–20% of patients who undergo knee replacement surgery in the NHS while having relatively poor health status.

The stage 2 analyses included a much smaller number of participants than stage 1: 2256 UKR patients and 57,682 TKR patients, of whom only 145 UKR patients and 23,344 TKR patients contributed primary outcome data. However, the safety analyses included almost 10 times more UKR patients and more than 200 times more TKR patients than TOPKAT.

Given that no RCT has included people comparable to the participants of UTMoSt stage 2, there are no gold standard data available for comparison. We, therefore, judged the performance of our analytical methods by their ability to minimise confounding for the variables available in our analytical data set. These analyses were still limited by the effect of potential unobserved confounders that were not recorded in any of the three linked data sources used for UTMoSt (NJR, HES and NHS PROMs database).

Of the tested methods, PS stratification based on the whole cohort most efficiently minimised confounding for the primary outcome analysis. PS stratification based on the UKR cohort and IPW led to unacceptable imbalances in some confounders and required double adjustment for the imbalanced variables. The safety analysis included a much larger population than the primary outcome analysis. The three methods successfully achieved balance for all of the known confounders available in the UTMoSt data set for this larger analysis.

When using the preferred method of PS stratification based on the UKR cohort, UKR had similar effectiveness (compared with TKR) to that observed in TOPKAT and UTMoSt stage 1, with an ATE of 1.83 (95% CI 0.10 to 3.56) OKS points in favour of UKR. Although statistically significant, this increased patient-reported benefit for UKR over TKR is not likely to be clinically relevant. After double adjustment for unresolved imbalances, PSS_whole yielded an estimate of 1.82 (95% CI 0.10 to 3.56) and an IPW estimate of 1.00 (95% CI –1.28 to 3.27). In summary, UKR had similar comparative effectiveness in patients with severe systemic disease and/or substantial functional limitations, as defined by an ASA grade of ≥ 3, as in the overall population. There were small, clinically irrelevant differences in postoperative OKS between UKR and TKR in these patients. Sensitivity analyses of effectiveness restricted to more experienced surgeons could not be performed because of limited statistical power.

Safety considerations, particularly short-term complications, are particularly important for patients with multimorbidity. As the two surgical approaches had no appreciable difference in benefit, any differences in risks would be highly relevant for decision-making. All three analyses suggested a strongly protective effect against postoperative venous thromboembolism for UKR patients, with a 60–67% relative reduction in risk compared with TKR patients. Venous thromboembolism is the most common postoperative complication of knee replacement and affected up to 8% of TKR patients and just below 3% of UKR patients in UTMoSt. Acute myocardial infarction and prosthetic joint infection were also analysed. In the first 90 days after surgery, almost 5% of TKR patients and just over 3.5% of UKR patients experienced myocardial infarction, and 1.9% of TKR patients and 1.8% of UKR patients experienced a prosthetic joint infection. However, no clear statistical difference for either complication could be demonstrated.

TOPKAT’s sample size did not give sufficient power to reliably study postoperative complications. However, our findings are consistent with a recent multinational collaboration study led by EHDEN (of which we are members) and OHDSI.¹⁶ For example, Burn et al.¹⁶ analysed over 32,000 UKR participants and more than 250,000 TKR participants after PS matching and found a 50% reduction in the risk of 90-day postoperative venous thromboembolism, but no significant reduction in the risk of infection. These results are also consistent with a meta-analysis published in BMJ in 2019,⁸⁸ which found that UKR was associated with a 60% reduction in the risk of postoperative venous thromboembolism compared with TKR.

The UTMoSt stage 2 results suggested that the relative effects of UKR and TKR on short-term postoperative complications in patients with severe systemic disease were consistent with those seen in previous literature for the wider population. UKR seemed to be safer in the short term and resulted in a lower (about 40–50% reduced) risk of postoperative venous thromboembolism in the 90 days after knee replacement surgery. This is highly relevant for patients and clinicians, as such complications can have deleterious effects on patients with baseline comorbidity.

We also assessed the effects of UKR (vs. TKR) on long-term consequences, 5-year revision surgery and mortality. UTMoSt stage 2 showed that, as expected, patients with complex health needs were more likely to die than have revision surgery. The 5-year cumulative mortality rate was 24% for UKR patients and 37% for TKR patients, compared with cumulative revision rates of 13% and 5%, respectively. Survival analyses found that UKR was associated with an almost threefold higher revision risk than TKR in these patients with systemic comorbidity, but with > 30% reduction in all-cause mortality. However, these analyses were hampered by a lack of reliable information on post-revision mortality, as the analyses were censored at the earliest of both events, and on cause of death. More data are, therefore, required to elucidate how UKR (vs. TKR) reduced long-term mortality and how the observed excess revision risk for UKR patients affects subsequent (post-revision) risk of death.

TOPKAT found that UKR had a lower health-care cost (to the NHS) and better cost-effectiveness at 5 years post surgery,⁵ with an additional benefit of 0.24 QALYs and a cost-saving of £910 per procedure over TKR. In UTMoSt stage 2, we used an economic evaluation to compare the cost-effectiveness of UKR and TKR for patients with severe systemic disease, as defined by an ASA grade of ≥ 3, who would have been excluded from TOPKAT. We used the three validated PS methods to minimise confounding, then analysed cost-effectiveness using similar health economics methods to those used in TOPKAT. These analyses demonstrated that UKR had an average cost of £6246 and TKR had a slightly higher average cost of £6627. Although UKR was associated with an increased revision risk, the cost of a revision after UKR was just over £5100, which was substantially lower than the cost of over £9100 associated with a revision after TKR. After discounting, UKR had a mean gain in quality of life within 5 years of 2.24 QALYs, higher than the 1.87 QALYs for TKR. The UTMoSt cost-effectiveness analysis suggested that UKR dominated TKR for patients with substantial comorbidity (ASA grade of 3 or 4), as it was more beneficial and less expensive.

The analyses performed in UTMoSt stage 2 were limited by the potential for residual confounding and information bias related to the use of data routinely collected for clinical purposes rather than for research purposes. However, it is unlikely that better data will be obtained from randomised studies any time soon. The striking finding that UKR was dominant over TKR for patients with multiple comorbidity should guide future provision of care in the NHS.

Public and patient involvement

A patient representative for the National Rheumatoid Arthritis Society was a co-applicant for the grant application. She assessed the study’s topic and relevance. She was involved in co-investigator meetings for the study, during which the study progress and results were assessed and evaluated. She has not raised any concerns about the work. We also had a plan to include a patient and public involvement (PPI) representative in the Study Steering Committee. However, despite a substantial effort by Versus Arthritis, no PPI representative was recruited.

Implications for future research and clinical practice

The results of UTMoSt stage 1 have clinical and methodological implications. Despite challenges inherent in the nature of the data used for these analyses, our findings suggest that real-world evidence and some PS methods can reliably mimic surgical RCTs. This is of fundamental importance and is very timely, as coming changes in the regulation of medical devices will probably require comprehensive observational post-marketing surveillance.

Key recommendations arising from stage 1 for future research include:

• PS stratification and IPW are useful for minimising confounding when evaluating the comparative effectiveness and risks of alternative medical devices and surgical procedures. PS matching, PS adjustment and IV analyses should be used with caution as they failed to replicate the RCT results in our study.
• More methodological research is needed to produce guidance on which analytical methods are preferable in different scenarios and circumstances for the post-marketing surveillance of medical devices and surgical epidemiology. Real-world evidence studies will continue to emerge in the absence of equivalent RCTs. It is likely that observational post-marketing safety studies will grow in number with upcoming European and global regulations, creating an urgent need for better guidance on the best use of analytical methods when evaluating surgery and implantable medical devices.
• Future attempts to mimic surgical trials should take into account the eligibility criteria of both patients and surgeons contributing to RCTs, as the patients and clinicians participating in RCTs are not representative of routine NHS care.

The main clinical implication of UTMoSt stage 1 arises from the finding that the potential additional benefits of UKR in terms of reduced risk and increased benefit are achieved only when performed by surgeons with high volume in this surgical technique. Health-care services, including the NHS, should consider centralising the delivery of UKR in specialised centres or by specialised surgeons to maximise the potential cost-effectiveness gains described in TOPKAT and consistently demonstrated in a sub-analysis of UTMoSt stage 1 restricted to surgeons with higher volume of knee replacement surgery.⁸⁹

UTMoSt stage 2 also has implications for clinical care. Although most surgeons support the use of UKR for fit young patients, our findings suggest that UKR has similar benefits over TKR for patients with severe systemic disease. We found that patients with severe systemic disease had better patient-reported outcomes (probably not clinically relevant) and dramatically fewer safety events, particularly thromboembolic events, after UKR than after TKR, as seen in fit young patients. Despite an excess revision risk, mortality was also lower among UKR patients. This information should be clearly communicated to patients. Patients with a limited lifespan might prefer a procedure that provides similar benefits but that is potentially safer in the short term, despite the fact that it may lead to higher risk of revision surgery in the long term.

From a NHS perspective, UKR is the preferable option for this patient subgroup where suitable, as it provides better and less expensive care than TKR. From NJR data, we estimate that about 50% of patients undergoing knee replacement would be suitable for UKR, but that < 10% receive it. More strikingly, less than 4% (2890/75,055) of patients with an ASA grade of ≥ 3 underwent UKR in our data set. There is a clear need for NICE guidelines on the use of UKR for patients with multimorbidity in need of knee replacement surgery.