Introduction
The economic impacts from medical research form a subset of many of the logic models presented in Chapter 3. A section of the literature has addressed the specific issues relating to undertaking exercises to determine economic impacts or the returns on investment from medical research and development spending. Previous reviews of the literature, which form a starting point for this review, have highlighted work that has been done to advance the field.2,205 The methods used to assess these impacts or returns on investment are born from the economic evaluation literature, and the difference in approaches lies largely in the scope of the cost and benefits assessed, and the valuation methods for seemingly non-monetary components of the impact. As stated in Chapter 2, the purpose of this review was, using Buxton et al.28 as a starting point, to identify studies since 2004 that have used any methods to attempt to value (in monetary terms) the benefits (health and cost savings) of a body of health research and link that with an investment in the body of research. Articles were included only if they contained a component that attempted to value the impact of research and development investment on population health.
The article in the Bulletin of the World Health Organization by Buxton et al.28 attempted to learn from previous studies that had estimated ‘monetary values for the societal benefits obtained from health research, especially those studies that have attempted to link (and value) benefits to a specific society from a specified (and costed) body of research’.
The authors characterised the identified methods into four categories:
valuing direct cost savings to the health-care system
valuing benefits to the economy from a healthy workforce
valuing benefits to the economy from commercial development
measuring the intrinsic value to society from health gain.
Studies were identified that had considered the benefit of medical research and development as direct cost savings to the health-care system, brought about by a reduced number of people requiring treatment or reductions in per patient treatment costs. This approach had been predominant in estimating the benefits of vaccination research, which had the potential to eradicate subsequent disease and associated treatment costs.205–208 Cost savings could be included as part of cost–benefit analysis, but these studies did not always link this to an investment period or country-specific research.
One of the earliest studies to attempt to calculate a rate of return from medical research was conducted by Mushkin and Landefeld.209 A human capital approach (equating the value of life to market values, i.e. wages) was used to value gains from US biomedical research, characterised by a healthier workforce. The limitations of such an approach were acknowledged by the authors and others28,209,210 and tend to overstate benefits when lost labour can be replaced, while understating benefits for those sections of the population not of working age.
Buxton et al.28 drew largely on a review conducted by Salter and Martin,211 which explored the commercial economic benefits from basic research. Salter and Martin noted progress made by Mansfield212,213 that estimated a worldwide social rate of return (benefits accrued to the whole of society, as opposed to one firm or funders of one project) of 28% for research undertaken 1975–78. Studies have also demonstrated the economic benefits of medical research through industrial applications to other industries.214
An emerging field highlighted by a number of studies in the Buxton et al.28 review had measured the intrinsic value of health gains brought about by research and development. A US initiative of the Mary Woodard Lasker Charitable Trust, Funding First,215 produced a series of papers that formed a subsequent book.216 An informal approach used willingness-to-pay methods to value the increased longevity of life experienced by the US population, attributing a fraction of these gains to medical research. The results suggested ‘exceptional returns’ of nearly 20 times the investment in US medical research. This type of analysis was performed in a more systematic fashion in an Australian study, taking a similar ‘top-down’ approach to valuing health gains, to produce an estimate of the annual rate of return to investment in research and development.217 They estimated a favourable benefit-to-cost ratio of 2.40 (i.e. AUS$1 invested creates an additional AUS$1.40 benefit); however, this work has been subsequently criticised because the time for investment in medical research to produce health gains was not considered.218
Buxton et al.28 noted that there is significant scope for these methods to be extended and refined to allow more robust estimation of the economic benefits from medical research. In particular, a widely acknowledged central challenge that must be addressed in this kind of analysis relates to the attribution problem; the relationship between investment in research and health outcomes.15,22,28,115 This manifests itself as several related issues regarding the contribution of health research in improving health outcomes and what would have happened without research, that is the unobservable counterfactual. Assumptions must be made regarding the share of health gains attributable to health research, and given there is an international pool of health research, the contribution of any particular country to particular health gains. Finally, assumptions must be made regarding the temporal relationship between a period of investment and a period of health gains. Different approaches face somewhat different problems in dealing with attribution, but methods have continued to be developed to address these issues.
Review findings
The search of databases produced 413 articles, which were initially screened by a reviewer by title (). After initial screening and deduplication, 102 articles were screened by abstract. Seventeen articles were reviewed in full, with five included.22,26,75,218,219 Two of these articles were included in the main literature review.26,75 One additional report that was not picked up by the supplementary search was included from the main literature review.25 One additional article and one report known to the authors was also added.24,220 One in-press article that the authors kindly gave us access to was also included.27 In total, the review produced nine articles/reports. The studies and methods of assessing return on investment, that included a component that attached a monetary value to health gains, are summarised in Table 16 (see Appendix 6).
Flow diagram of included studies.
The nine studies identified in the review can be split into two categories in terms of how health gains are measured: those taking a top-down approach and those taking a bottom-up approach. There are several other important issues that must be addressed in quantifying the returns; this simple taxonomy allows us to explore the different methodologies. depicts the basic methodologies.
Approaches to identifying health gains from research. R&D, research and development.
Studies have been compared on a number of key facets of the analysis and assumptions that have to be made regarding measuring and valuing net health gains and how to attribute a proportion of health gains to a body of research as follows: How were health gains measured? How were health gains valued? Were health gains ex post or ex ante? Were the costs of delivery accounted for? Was the lag between investment and health gain considered? How was the attribution problem addressed?
Top down
A stream of work undertaken by Access Economics (now Deloitte Access Economics) assessed the benefits of medical research in terms of the intrinsic value of the health gains to society. Two studies were conducted to estimate the returns on investment from Australian research and development.24,25 Access Economics considered all Australian health research and development spending both public and private between 1992 and 2005.24 Building on their approach in an earlier report,217 they used projections from the Australian Institute of Health and Welfare to estimate DALYs averted in the period 2033–45 relative to 1993 levels and calculate a return on investment of 117%.24 The authors assume that the lag between investment and realisation of health gains is 40 years, although the rationale for this figure is unclear.
To calculate the return on investment the authors considered the proportion of DALYs averted attributable to research and development, as opposed to other factors claimed not to be a result of research and development. The authors state that other factors include ‘public health awareness and preventive programs such as ‘Slip Slop Slap’ or ‘Quit’, screening and early intervention initiatives, the public subsidy of drugs and interventions through the Pharmaceutical Benefits Scheme and the Medicare Benefits Schedule, and so on’.24 The extent to which these examples are not research and development-based interventions could be heavily debated, especially screening programmes; however, the premise that external factors other than research and development are responsible for health gain has been widely acknowledged.221 They attributed 50% of health gains to research, as they had in their previous study, but have acknowledged that this was not robust.217 The return was highly sensitive to the value of this parameter. The authors take account of research and development conducted in others countries and its contribution to Australian health gains by using bibliometric techniques to estimate a proxy, based on Australia’s share of publications in the clinical sciences. They estimated that 3% of health gains could be attributed to Australian research and development. The DALYs averted were monetised using a willingness-to-pay methodology, attaching the value of a statistical life-year [AUS$266,843 – £124,300 (converted at 2015 purchasing power parity exchange rate)].222
A further study estimated the returns from National Health and Medical Research Council (NHMRC) funding in five disease areas (cardiovascular disease, cancer, sudden infant death syndrome, asthma and muscular dystrophy) using the same methodology.25 They estimated that the return on investment ranged from 509% in cardiovascular research to –30% for muscular dystrophy. These returns also included the value of avoiding direct health system expenditure, the value of avoiding indirect costs (through productivity losses), the value of direct commercial gains from the NHMRC-funded research and development, and benefits of NHMRC-supported commercialisation. Neither of the Deloitte Access Economics studies considered potential increased costs borne by the health-care system from expensive new technologies.24,25
Health gains were measured using DALYs averted in 2040–50 relative to 2000 levels.25 The time between investment (2000–10) and health gains (2040–50) was again assumed to be 40 years. It was assumed that the proportion of gains as a result of research and development was 50%, and 3.14% of these gains were assumed to be attributable to Australian research and development (re-estimated using bibliometric techniques). However, the authors were presented with an additional necessary estimation; the proportion of health gains that were a result of this programme of NHMRC research, rather than the whole body of Australian health research and development. Using bibliometric techniques they found that 25.04% of Australian research publications were funded through the NHMRC and used this as a proxy. The DALYs averted were monetised using a willingness-to-pay methodology (based on individual’s valuation of avoiding mortality/morbidity) attaching a value of a statistical life-year [AUS$168,166 – £78,300 (converted at 2015 purchasing power parity exchange rate)].222
Roback et al.218 used a broadly similar approach to value Swedish gains from all public and private research and development spending on clinical and health research in the year 2005. In this tentative modelling exercise, average annual increases in life expectancy (population utility adjusted) were used to estimate QALY gains in 2015. This implies a lag of 10 years, but the authors did not explicitly discuss this. QALY gains were valued using the value of a statistical life-year [SEK500,000 – £37,900 (converted at 2015 purchasing power parity exchange rate)].222 The returns were estimated at a socioeconomic level, including a whole range of non-health benefits where they could be quantified, resulting in a return on investment of 1.08 (8%). In making this estimate, the authors did account for ‘more expensive healthcare due to new methods’. They assumed that 50% of health gains were attributable to research and development, referencing estimates made by various authors that suggest the range may be between 25% and 67%.216,223–225 The proportion of health gains attributable to Swedish research and development was assumed to be 3% based loosely on an estimate of Sweden’s share of global expenditure and global medical publications.
A significant drawback of Roback et al.218 and the Access Economics24,25,217 work is the ex ante nature of health gains: the reliance on predictions based on previous trends in population health improvement. This assumes the impact of as of yet unobserved future usage of interventions and hence improvements in health. Ex post studies use retrospective data, either by directly observing population health gains or by compiling data on observed uptake and modelled per patient incremental net health benefits. Although many of these studies require pragmatism in assumption making, the reliance on unknown unknowns requires a leap of faith.24,25,217,218
Lakdawalla et al.220 assessed the social surplus arising from the ‘war on cancer’ in the USA from all public cancer research and development spending between 1971 and 2000. An upper bound of this investment was estimated to be US$300B, based on National Cancer Institute spending (which was assumed to make up approximately one-quarter of cancer research and development spending). Ex post life-year gains in survival between 1988 and 2000 were identified and valued at individual willingness-to-pay [US$30,737 – £21,300 (converted at 2015 purchasing power parity exchange rate)].222 This produced an estimate of the net gains at US$1.6T. Lakdawalla et al acknowledge the likely lag between investment and health gain and suggest that they may have overestimated the size of investment and hence conservatively estimate social surplus, but did not explicitly investigate the lag. The survival gains were estimated based on cancer-specific improvements in detection and treatment, although the potential for non-research and development contributions to these improvements was not considered.
Bottom up
Informed by methodological frameworks, such as the Payback Framework, studies have used a different approach to build the benefits up from individual interventions to estimate the sum of the health gains, rather than starting from an estimate of overall health gains.39 It, in part, theoretically deals with the attribution problem presented when trying to estimate the contributions of research and development and non-research and development factors in producing health gains, although it produces a different challenge in identifying only those interventions that are known to have been research driven.
Johnston et al.75 applied such an approach to the US National Institute of Neurological Disorders and Stroke’s funding of 28 Phase III RCTs prior to 2000. They estimated a return on investment of 46% per year based on 10-year estimates of post-funding QALYs. Available cost–utility analyses were used to estimate the per-patient QALY gains for eight interventions, and data on use were gathered to estimate population gains. Implicitly, it was assumed that all changes in use post trial were a result of that clinical research. Although the examination of the use of the eight interventions suggests some lag, with use fairly stable for at least 2 years after the completion of funding, it might be considered shorter than other estimates.226 Data presented by the authors suggest that use is not zero during the period when funding ends, which might be indicative that other research not funded by the National Institute of Neurological Disorders and Stroke could have played a role in health gains. By using cost per QALY utility data, the authors were able to present monetised health gains [valued at GDP per head of US$40,310 – £27,900 (converted at 2015 purchasing power parity exchange rate)]222 net of costs of delivery (net monetary benefits) for each intervention. The study was able to find adequate data for only 8 of the 28 Phase III trials, which highlights the data-heavy nature of this exercise and the reliance on published literature. In some instances, a paucity of data may limit the ability for such a study to be undertaken or at least limit the generalisability of findings.
Two studies published by authors of the Payback Framework have adopted an approach that is similar with respect to the identification of health gains to the work of Johnston et al.,75 but have focused on quantifying the returns in different disease areas.22,26
Buxton et al.22 estimated the return on investment [presented as an internal rate of return (IRR) that considers the flow of cost and benefits] from publicly and charitably funded cardiovascular research in the UK to be 9% per year (£1 investment yields health gains equivalent to £1.09). They estimated the health gains between 1986 and 2005 and linked this with a period of investment between 1975 and 1988, based on a lag of 17 years. The lag was estimated based on citation analysis of UK guidelines, using mean time between citation and guideline publication (‘knowledge cycle time’) as a proxy for the time between investment and health gain. Research-led interventions in the cardiovascular field were identified and a timeline of usage assembled. For each of the interventions, per-patient QALY gains and net costs (increases from delivery and potential savings from reduced sequelae) were identified through published cost–utility analyses. QALY gains were valued at the health-care service opportunity cost based on implied cost-effectiveness thresholds of NICE (£25,000) and presented net of costs, to produce an estimate of the net monetary benefits produced per year. The NICE threshold value was chosen to reflect the competing nature of funding of health research over provision of existing technologies. It was assumed that 17% of the health gain was attributable to UK research, based on bibliometric analysis of cardiovascular guidelines that identified the proportion of cited work that contained a UK corresponding author. Buxton et al.22 combined this IRR with the wider GDP spill over effects of research and development, estimated to be 30%, to give an overall IRR of 39%.
Glover et al.26 applied the same methodology to publicly and charitably funded cancer research in the UK, re-estimating the lag between investment and health gain and the proportion of health gains attributable to UK research based on cancer guidelines. An IRR of 10% was estimated, based on the monetised net health gains for 1991–2010 for research-driven interventions, linked to cancer funding between 1976 and 1995 (15-year lag). This work highlighted the difficulty in identifying all of the important research-driven interventions. An additional publication227 used accompanying case studies to highlight the complex and heterogeneous relationship between research and health gains. There is a need in a field such as cancer to narrow the scope to complete such a resource-intensive exercise, where there have been widespread improvements in detection and treatment brought about by research, and where the benefits are realised across a heterogeneous patient population (for instance there are over 200 types of cancer). Although developing a method that used changes in incidence and survival gains as a predictor of which cancer types were likely to have contributed largely to overall gains, the authors assumed that interventions not represented in the analysis produced zero net benefit.
A study by de Oliveira et al.219 largely replicated the methods presented in Buxton et al.22 to assess the return from Canadian publicly and charitably funded cardiovascular research, which they estimated to be 21% per year based on QALY gains in 1994–2005. Using similar bibliometric techniques, a time lag of 13 years was estimated and 6% of overall health gains were attributed to Canadian research and development. They also argued that an additional component should be considered as part of the attribution problem, assuming that 70% of the health gains were attributable to medical research. However, if the identified interventions were research led and studies used to estimate per patient health gains produced incremental differences brought about by the specific intervention, it is not clear why non-research and development factors ought to be considered in this context.
Guthrie et al.27 estimated the benefits of the NIHR HTA programme funding from 1993 to 2013. They selected 10 key HTA studies, which were largely made up of randomised trials but also systematic reviews. They identified the per-patient QALY gains associated with the interventions. QALY gains were monetised at the health-care opportunity cost (£20,000 and £30,000) net of health service costs, but total actualised gains were not estimated. Instead, a net monetary benefit associated with a hypothetical 1 year of full implementation for the patient population of the interventions was calculated; therefore, the lag between investment and gains was not considered. The HTA studies were considered to be responsible for all post-HTA research implementation, as they were seen to constitute ‘definitive’ evidence. The authors suggest that only 12% of potential net benefit would cover the £367M invested by the NIHR HTA programme. Although indicative of potential gains, this analysis does not adequately address the attribution problem and makes no consideration of when benefits accrue. It also raises the interesting problems posed when the research takes the form of systematic reviews and the role of such a study in changing clinical practice and hence leading to health gain.
Discussion
There have been contributions to the literature that estimate the impacts of health research using methods to attach a monetary value to health gains. Approaches have attempted to estimate the resultant health gains from investment in bodies of research, and, in doing so, must deal with several problems relating to attributing health gains to particular investments. Techniques that attempt to deal with the problems of attribution have been established. However, authors have acknowledged a simplification of the relationship that is required and the reliance on a logic model view of research impacts. Some of these contributions also consider non-health sector benefits falling on the wider economy, although the scope of the benefits considered often differs, as does the valuation.
Only a few studies specifically considered programmes of health research.25,27,75 Guthrie et al.27 estimated the gains of the NIHR HTA programme, but made cautious conclusions on the returns based on hypothetical uptake of a subset of HTA-funded research.27 Clearly there is scope for these types of methods to be applied to estimate returns from programmes such as NIHR HTA, but several additional considerations need to be taken into account. Conversely, there are advantages to having a well-defined unit of analysis.
It would appear that assessing monetised impact at a programme level is conducive to the bottom-up approach, when the set of interventions is well defined and the task of identifying those that are ‘important’ could be avoided. To an extent, data feasibility issues that limit the bottom-up approach should be mitigated by programmes such as NIHR HTA in that most of its research includes cost–utility estimates. However, issues of scale are present if the number of studies undertaken by a programme is large, such as in the NIHR HTA programme. This might be mitigated to an extent by the need to consider only those trials that showed a significant effect, but this makes a bold assumption about the nature of evidence being used in clinical practice.
When attempting to measure health gains from a programme of research, the attribution problem manifests itself as an added layer of uncertainty regarding the proportion of total health gains that should be attributed to the specific programme. Using a top-down approach, Deloitte Access Economics25 dealt with this by using the percentage of total citations in clinical sciences that were studies funded by the programme as a proxy. This additional attribution problem is not circumvented by the bottom-up approach and a consideration must be still be made. The view taken in Johnston et al.75 and Guthrie et al.,27 that National Institute of Neurological Disorders and Stroke’s trials and HTA studies are definitive in terms of changes in uptake, is insufficient, especially in developed countries with multiple funding streams and complex and evolved research ecosystems. The use of the weights attached to particular RCTs in meta-analyses could provide a more systematic way of considering the relative impact of different clinical research supported by multiple funders. Regarding the proportion of health gains that should be attributed to world research and development, this could be used as an intervention-specific replacement for guideline analysis or used as an adjunct. Although time lags must be included, it is not clear how best to estimate these.
An additional problem for programmes that fund only clinical research is dealing with the role of basic research in health gains. This is unclear and constitutes a major potential limitation in these methods. No study looking at a programme has yet encompassed these kinds of considerations into the approach dealing with attribution of health gains to a programme, using either top-down or bottom-up methods.
Although it would clearly be possible to estimate the returns on investment from the NIHR HTA programme, significant challenges remain.
