NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
National Academies (US) Committee on Measuring Economic and Other Returns on Federal Research Investments. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington (DC): National Academies Press (US); 2011.
Measuring the Impacts of Federal Investments in Research: A Workshop Summary.
Show detailsThe impacts of research on the health of people in the United States and around the world may not be measured by economic analyses, but historically these impacts have been among the most important benefits of research. Five speakers with diverse backgrounds addressed this topic at the workshop. They found evidence of substantial benefits while also identifying areas where benefits may be overlooked by current approaches. In addition, they called attention to problems with the funding of federal research, such as the damage up-and-down funding can do to the careers of researchers and the difficulty of allocating limited funds across categories of research.
REVIEWING THE LITERATURE ON HEALTH IMPACTS
Bhaven Sampat, Assistant Professor of Public Health at Columbia University, presented a brief summary of a commissioned paper (Appendix D) that discusses representative studies of the effects of publicly funded biomedical research on a range of outcomes. Public funding accounts for about one-third of all biomedical and health research, with NIH-sponsored research accounting for most of the federal component along with additional investments by NSF, DOE, DOD, USDA, and other agencies. In 2007, funding for biomedical research totaled slightly more than $100 billion.
Sampat showed a stylized albeit simplified view of the innovation system in which publicly funded R and D leads to improvements and efficiencies in the private sector, to new drugs and devices, and ideally to improved health outcomes (see Appendix D, Figure D-1). This flow of knowledge occurs through many channels. One channel encompasses publications, conference presentations, markets, and informal networks. A second channel is through the creation of prototypes for drugs and devices. Since the Bayh-Dole Act of 1980, these prototypes have tended to be developed in universities and licensed out to firms to turn them into successful products. A third channel includes funding for clinical trials and clinical research that informs clinical practice— such as the knowledge that doctors should give people an aspirin after a heart attack — along with funding of other applications-oriented work, such as contracts to fund the development of technologies and to conduct consensus conferences.
Sampat called attention to another impact of new biomedical technologies that is being discussed among health policy researchers. Most economists believe that biomedical technologies are the biggest source of long-run increases in health care costs. The clinical value from these technologies may exceed their costs, but technology-driven cost increases may be unsustainable, Sampat observed.
The Case of Cardiovascular Disease
Sampat described some of the literature on improvements in health outcomes that can be traced to research. Cutler and Kadiyala (2007) looked at improvements in cardiovascular disease mortality over the five decades beginning in 1950, when mortality fell by two-thirds. They concluded that about one-third of the advance is attributable to new high-technology treatments, one-third to new drugs, and one-third to behavioral changes such as not smoking and not eating salty or fatty foods. Using a standard evaluation of $100,000 per year of life used by health economists, they then computed the rate of return on investments in treatments. New treatments provided a 4-to-1 rate of return, while new behavioral knowledge produced a 30-to-1 rate of return. According to this paper, Sampat said, “the publicly funded R in R and D has been worth it.”
This paper makes little mention of NIH or public research except for NIH’s sponsorship of large epidemiological trials and conferences, which makes it hard to trace outcomes back to basic research. Another issue, said Sampat, is the counterfactual: What would have happened in cardiovascular disease absent any public funding in that area?
A paper by Heidenreich and McClellan (2007) focused on improvements in heart attack care. These authors go farther than Cutler and Kadiyala in relating changes in clinical practice to specific outputs of R and D. The authors concluded that the medical treatments studied in clinical trials accounted for much of the improvement in heart attack outcomes. The challenges with this paper include the fact that the authors generally did not trace changes in clinical practice back to basic research. Also, clinical practice often leads to publicly funded R and D because informal learning by clinicians generates important research questions. This learning is often subsidized by Medicare payments to teaching hospitals and other non-research sources. Finally, clinical trials can lead to negative results and lead clinicians to stop doing things they were doing, which can be an unmeasured benefit to research.
Other Disease Categories
A statistical study by Manton et al. (2009) related mortality rates in four disease areas to lagged NIH funding for the relevant institutes from 1954 to 2004. For two of the diseases studied—heart disease and stroke — the authors found a relationship between funding and outcomes. For the two other diseases—cancer and diabetes - the evidence was weaker. But relying on funding aggregated by institute is difficult, as an institute can fund widely varying research. Also the counterfactual is hard to demonstrate since many factors could be driving changes in disease rates.
Over this time period, competing risks changed. One reason for the absence of a decline in cancer mortality— and maybe even an increase—is that fewer people are dying of heart disease, so they live longer and are more likely to develop cancer.
Relationship of Public and Private R and D
Papers by Toole (2007) and by Ward and Dranove (1995) sought to relate public sector R and D to private sector R and D and found strong evidence that they are complements rather than substitutes, in that public research tends to spur private research. Private and public R and D in a given area could be driven by scientific opportunity. For both forms of research, there are challenges linking R and D to health outcomes, Sampat observed.
Another line of research regarding private sector R and D is whether proximity to public sector scientists makes firms more productive. A range of studies have indicated that the answer is probably “yes,” Sampat said, especially survey research asking firm R and D managers how much they rely on public sector R and D.
Sampat noted that in surveys the drug industry reports greater reliance on public sector R and D than do other industries. In contrast, the device industry tends to be at or below the mean in terms of reliance on public sector R and D. The drug industry relies mostly on medicine, biology and chemistry. The device industry relies on medicine and biology and, third, on materials science, which tends to be funded by NSF and DOD.
Sampat then turned to drug and device innovation. Very recent studies have used accounting methodologies to look at, for example, the impact of public sector R and D in producing drugs that are then marketed. In a study of drugs in FDA’s Orange Book, about 10 percent of marketed drugs come from universities or public laboratories, meaning that these institutions hold key patents (Sampat and Lichtenberg, 2011). The number is higher, about 20 percent, for clinically important drugs.
The Case of HIV Drugs and Vaccines
HIV is a special case, Sampat observed. The role of the public sector in directly generating new drugs is much higher in HIV than in other arenas; nearly one third of drugs in this area rely on public sector research. Also, nearly all commercially and therapeutically important vaccines over the last 25 years have come from the public sector, according to Stevens et al. (2011). Surprisingly, efforts to relate funding by disease area to later drug innovation tend not to show much of an effect.
Device Development
In the areas of devices, Sampat described a case study by Morlacchi and Nelson (2011) on the development of the left-ventricular assist device (LVAD). The scientific understanding of heart failure remained quite weak throughout the period that the LVAD was developed. But NIH was holding consensus conferences to diffuse best practice and contracting with firms for device development and clinical trials. “The more applied side of the activities seems to be important” in this case, Sampat concluded.
Conclusion
The literature shows “consistent evidence of public sector funding on private sector innovative effort,” Sampat concluded. The literature also shows that public sector R and D has been important in the generation of a non-trivial number of important drugs. However, it shows less impact on other innovative outputs.
“There is surprisingly little research on the health benefits of public sector biomedical R and D,” Sampat observed. Most of the evidence to date is from the cardiovascular area. In addition, case studies point to the importance of public clinical research, applied research, and diffusion activities. Devices have important differences from drugs. And despite a good deal of discussion, there has not been much study of the effects of public sector research on health costs.
THE VOLATILITY OF FEDERAL R AND D SUPPORT
Richard Freeman, Herbert Ascherman Chair in Economics at Harvard University, addressed the unintended effects of variability in federal government funding for R and D. Using changes in the budgets of the National Institutes of Health as an example, Freeman said that chief among these effects is the damage done to people’s careers by changes in grant rejection rates and increased uncertainty about future career prospects. Scientific careers “looked dicey” even after the Wall Street implosion and lay-offs in banking and consulting made finance less attractive.
Funding variability may also affect the productivity of scientific research. His study of the recent doubling of the NIH budget found that before the doubling period more papers were produced per dollar of grant than when more money was available. This decline in marginal productivity may make it easier to cut future funding due to “failing to meet ‘promises,’” Freeman stated. By contrast, the private sector has not been as variable in its R and D support.
Gaps in Monitoring Science
Finally, Freeman suggested that the scientific community needs to do a better job of monitoring the state of science. For example, non‐traditional measures of the supply of jobs might include real-time data from Internet job boards, searches for information about science and engineering jobs, and databases on Ph.D. dissertations. Downloads of working papers could indicate hot areas of research. Online science and social discussion groups and web-based communications from meetings and conferences could contain information useful to the policy community. Companies and other institutions should be accessing these databases regularly, he said.
Information on what industry is doing is weak. The aggregate amounts of money spent do little to map the steps to innovation. Even the NSF BRDIS survey provides little data beyond the amounts of money spent. Further, basic and applied research tend to be artificially divided, but anything that is an innovation is going to go back and forth between the two categories of research, Freeman observed.
MEDICAL DEVICE INNOVATION
Many people assume that the development of biomedical devices is similar to drugs, said Paul Citron, retired Vice-President at Medtronic, Inc., and now at the University of California, San Diego; but in fact “they have very different characteristics as they traverse the pathway from bench to bedside.” Drugs tend to be more discovery-based and derived from in-house activity. Devices are engineering-based. A specification is generated, along with an idea of how to realize that specification. Moreover, devices evolve over time. The first device is very different from subsequent generations, whereas a drug tends to be static for its lifetime.
For devices, the timelines are longer and the markets are smaller than in the pharmaceutical industry, Citron explained. It is very rare for a medical device to have a billion dollar market, unlike pharmaceuticals.
The evolution of a device can be heavily influenced by federally funded research, according to Citron. For example, research can enable an industry to bring a device from concept to clinic. Federal funding can build the underlying knowledge needed to make a technology safe and effective. Federal research also can yield new materials, whereas the complexity and cost of coming up with a new biomaterial to be implanted in the human body can be beyond the ability of any one company. Clinical trials may be crucial in improving a device.
A successful outcome for interventions using a medical device depends on rigorous manufacturing, which can be improved through R and D cycles involving federal research. Most medical product recalls are due to manufacturing issues that arise after approval, which can be reduced through R and D.
Vivariums at academic centers are another crucial investment underwritten by federal support. Prototype products are often tested at these institutions, and even large companies may need to use academic centers for access to animals.
Finally, “probably the most important output of federal inputs,” said Citron, is students. “We hire the products of the campus” because “that is where the intellectual horsepower for tomorrow resides.”
Citron listed four criteria an industry uses to decide whether to pursue a project. (1) Does the technology fit with a company’s internal capabilities? (2) Is the fit with the customer good? (3) What is the market opportunity, including the number of customers, price, and the details of application? (4) Finally, what is the time to market, including the time needed to satisfy the regulatory process?
MAKING DECISIONS IN THE PHARMACEUTICAL INDUSTRY
The pharmaceutical industry and regulatory bodies need to evaluate drugs thoroughly and expeditiously as they go through years of clinical development before gaining approval for use in the treatment of a particular disease state, observed Dr. Garry Neil, Corporate Vice President for Science and Technology at Johnson and Johnson. Dr. Neil’s company discovers and develops therapeutic products and technologies that are evaluated by regulatory agencies around the world to assess the efficacy and safety of a product for its intended use. “We have set the bar very high for ourselves [about] what is expected and what we need to deliver to our stakeholders, and we take that very seriously.”
Drug discovery spans years of study or phases of study, from prediscovery to post-marketing surveillance, which allows for continued follow up in a real-world setting after a therapy had been approved – but getting to the point of approval can be challenging. Typically thousands of compounds are synthesized to yield just a few potential candidates that enter preclinical study, and for every five thousand to ten thousand synthesized compounds, one approved drug on average may emerge. And for many reasons, the costs to develop new drugs have risen precipitously, which is further complicated by the fact that fewer drugs are commercially successful “Despite all this, we continue to press very hard because we recognize that there is unmet need and there are financial rewards for real innovation that can really help people, even if it’s the exception rather than the rule,” said Neil.
In recent years, public confidence in the pharmaceutical industry and in the regulatory system has eroded. This may have the effect, if the regulatory process is lengthened, of delaying the introduction of innovative products or adding additional expense to the process and ultimately the final approved product or medicine. “We can’t sacrifice rigor, and no one is suggesting that, but we need to recognize the consequences of raising the regulatory bar.”
To improve both productivity and regulatory certainty, said Neil, work needs to continue on understanding basic biology. “It’s not easy,” he said. “This is going to require a lot of collaboration between industry and academia.” In addition, a new tool set is needed for drug discovery and development as it relates to translational medicine, and these tools need to be customized for particular diseases to increase the likelihood of an efficacious therapeutic agent for a particular disease.
The United States should invest in an infrastructure akin to the Internet or the interstate highway system in which it would be possible to enroll patients in clinical trials much more rapidly, whether for drug trials, observational studies, investigations of medical devices, or other research. Only 3 percent of cancer patients enroll in clinical trials today. “We make it inconvenient for them. Do we need an institutional review board in every university? Why can’t we have national review boards? Why can’t we have national safety monitoring committees? Why can’t we bring the cost down and make the efficiency much better? Why can’t we include patients of color, women, and older people? We’re not getting those people today.”
The nation needs a more sophisticated and effective safety and performance monitoring system for drugs once they enter the market. And, most important, said Neil, health care providers need a system to provide them with the latest information at the point of care to help them make the best possible decisions for each individual patient.
FDA regulates 25 percent of the U.S. economy, representing over $1 trillion worth of spending and a third of all the imports, with just 11,000 people and a $3 billion budget. “They need help,” said Neil, including contemporary tools and techniques for pre- and post-marketing evaluation. They also need new risk assessment tools and much better engagement of patient communities.
“The standard way of looking at this is to talk about risk and benefit,” said Neil. “What is the benefit of the treatment? What is the risk? . . . I think a better way of looking at this is risk and risk. There is a risk of not treating a disease. What is that risk? Then there is a risk of treating the disease. What is that risk, and what does that risk ratio mean in the minds of the patient?”
RESEARCH AND OUTCOMES CASE STUDY: PEDIATRIC HIV
Laura Guay, Research Professor at the George Washington University School of Public Health and Health Services, provided a perspective on research funding and evaluation by a philanthropic foundation. As vice president for research, she spoke about the work of the Elizabeth Glaser Pediatric AIDS Foundation, founded in 1988 to prevent HIV infection and eliminate AIDS among children in the United States and abroad through research, advocacy, and treatment programs.
Early on, the foundation studied how children are infected, how many children are infected, and why children are infected, chiefly through “scientist awards” to encourage young investigators to develop their careers in this less known field. The awards have provided $750,000 to individual investigators for capacity building rather than for specific research questions. Since 2007 the foundation also has made operations research grants to improve treatment program design and scale-up. As is often the case with medical research, there are obstacles to the delivery of science into the field, especially in developing countries, Guay noted.
Guay said that the foundation chooses innovative studies that are less likely to be funded through NIH. “Why isn’t this fundable by the NIH” is one question on its application. For example, while funding for HIV vaccine-related studies is plentiful, very few of these funds focus on a vaccine in infants born to breast-feeding mothers. The foundation also may fund young investigators who do not have sufficient credentials to compete successfully for NIH grants.
To measure the impact of its research investments, the foundation needs performance metrics for deciding the impact of that funding, Guay observed. For example, an important question has been how awards have leveraged additional funds. Dating from the first funding of scientists in 1996, the foundation identified early leading scientists, which has generated an “exponential increase” of originally small investments over time.
Guay described two examples of the foundation’s operations research, both influential in improving maternal HIV diagnosis and antiretroviral treatment in African countries. The first involved a controlled experiment in training nurses in the appropriate follow-up to a positive diagnosis of the infection. The second involved an experiment in rapid syphilis testing in connection with rapid HIV testing. Both projects provided evidence for methods of identifying more infected women, preventing transmission to their babies, as well as attracting men for testing and treatment.
Guay concluded her remarks with an illustration of successful application of research results that was recalled several times later in the workshop (Figure 4-1). In 1994, when research results showed that treatment of pregnant women with antiretroviral drugs could prevent babies from being born infected with HIV, the number of perinatally acquired AIDS cases dropped from approximately 900 per year in the United States to virtually zero over the next decade and a half. This dramatic outcome depended on progress dating to well before that date in human capacity, laboratory capacity, and clinical capacity, Guay observed. It is important to consider “all of the pieces that had to be in place” as “we continue to eliminate pediatric HIV in the rest of the world.”
FIGURE 4-1
After it was shown that treatment of pregnant women with antiretroviral drugs could prevent babies from being born infected with HIV, the number of perinatally acquired AIDS cases in the United States and dependent areas dropped precipitously. SOURCE: (more...)
Sampat asked how the Glaser Foundation allocates funds among basic studies, vaccine development, and operations research, and Guay said that in the early years the foundation considered its funds unrestricted. But as more work has been funded by NIH and others, the trend has been to “donor-driven” funding for particular projects or areas. Because “people believe NIH has a lot of money,” it is harder to raise foundation funds for basic research. And the biggest challenge, said Guay, is “we have a lot of science we haven’t figured out how to deliver.”
DISCUSSION
Given the “exceptional return” of HIV-AIDS research in the United States, “we need more case studies on failures” to figure out why some research avenues have not been more productive, said Sampat. Also, in evaluating the outcomes of public sector health research, it is hard to aggregate across disease areas. Sampat cited the Research, Condition, and Disease Categorization (RCDC) database started by NIH in 2009, which reports on 229 diseases and research areas of interest to Congress, but “these are not necessarily the diseases of historical interest to economists and policy analysts.”
Kai Lee of the Packard Foundation, who spoke later in the workshop, asked if the data show “there is a lot more to be gained in the biomedical field from behavior-focused research?” He noted that Freeman, Citron, and Guay had all suggested the importance of human and institutional elements to outcomes. Freeman agreed that institutional and behavioral factors are important in the environmental area; they appear frequently on NIH’s list of grand challenges as well. For example, the biggest success in preventing cancer has been behavioral, with regulatory, marketing, and other factors all working to reduce smoking.
Citron pointed out that that over time the optimal ratio of biomedical research to behavioral change could change. For example, though cigarette smoking has declined, many aspects of diet still need to change to improve health.
- IMPACTS ON BIOMEDICAL AND HEALTH RESEARCH - Measuring the Impacts of Federal Inv...IMPACTS ON BIOMEDICAL AND HEALTH RESEARCH - Measuring the Impacts of Federal Investments in Research
Your browsing activity is empty.
Activity recording is turned off.
See more...