PITFALLS ON THE ROAD TO UNDERSTANDING

• The selection of specific measures inevitably focuses attention and effort on what is being measured. Their value can decay as more of what is being measured is produced. Further, the selection of metrics can reduce the valuable diversity of the research system and its potential for serendipity.
• Agricultural research has been so successful that Americans now pay less for food per capita than in almost any other country and any other time. This success may have had the perverse effect of undermining funding for basic agricultural science, since the need for productivity gains seems less pressing.
• Research funding volatility has major consequences for the decisions made by research performers. For example, the doubling of the NIH budget drove a large expansion of biomedical research facilities at research universities in the expectation that increases would continue. The suspension of real growth at NIH halted the growth of indirect cost recovery to pay for those buildings, with adverse effects for other parts of the university. Funding patterns also send messages to students about desirable fields of research – messages that may be at odds with long-term employment prospects in those fields. Volatility is problematic in firms as well as in federal research.
• An assessment of even a narrow field requires taking an average from disparate processes and systems, which can cause such assessments to be overly broad. For example, the number of patents granted within a particular field may be important, but individual researchers should not be judged by how many patents each one has generated.
• If all past research had been required to justify its value in terms of practical benefits, advances that have led to massive practical benefits would not have occurred.
• The knowledge generated by fundamental research has an intrinsic value regardless of its application. Without it, applied work would stagnate.
• Policymakers and the public in general agree on the value of research. Could research that fails to identify many of the benefits of science undermine that consensus and therefore be harmful?

PROGRESS IN UNDERSTANDING THE ISSUES

• The ever-growing power of the Web and the information sharing it enables will facilitate the analysis of research outputs. Natural language processing, machine learning technologies, and crowd sourcing will increasingly glean many reasonably accurate metrics from publications, patents, social networks, blogs, and so forth, and this capability will increase over time. Furthermore, this approach will be less costly and provide more information than government-mandated reporting. However, government agencies will need to create new tools to use these data to help fulfill their missions.
• The benefits of research results, both in terms of new knowledge and trained students, are vastly different from discipline to discipline and even from subdiscipline to subdiscipline. Thus, the determination of impacts requires very detailed analysis that is highly sector specific. For example, the evaluation of physics is different than the evaluation of computer science, and the evaluation of theoretical computer science is different than the evaluation of research in parallel computation.
• Relatively small expenditures on increasing the dissemination of research results could greatly enhance the beneficial impacts of research without entailing major new funding initiatives.
• Some questions may not be answerable, and identifying those questions may usefully focus attention on the questions that can be answered.
• The major discoveries that result from basic research are outliers that generally are very hard to predict. They emerge rarely, but they are the most important. How can these be accommodated in assessments of the value of research?
• As science becomes more interdisciplinary, more collaborative, more international, more digital, more open, more expensive, more diverse, and more fast-paced, measuring impacts will face new and difficult challenges.

OPPORTUNITIES POSED BY GREATER UNDERSTANDING

• The science of science policy has an opportunity to examine the broader issues of economic growth and societal change if it interprets its agenda broadly. As an example of an important albeit difficult question, are additional funds most usefully spent on health-related R and D or on insurance? Some analysts have cited the drop in deaths from cardiovascular disease starting in 1965 as an outcome of biomedical research, but that was also the year when Medicare was instituted.
• The plural of anecdotes may not be data, but anecdotes can be more powerful than data in swaying policymakers, even if they are not necessarily representative.
• A heightened emphasis on accountability within government will increase the need to produce metrics of research impacts. The research community needs to understand why this is important, especially because they can contribute ideas that would benefit data collection and analysis.
• Research funders and performers have many opportunities to work with the private sector in measuring the impacts of research, since the private sector spends considerable time and money working on this issue.
• The optimal amount of research for the United States as a percentage of GDP still has not been determined. Is it possible to overspend on R and D? To what extent should education be emphasized in that spending?

CONCLUDING REMARKS

Finally, several speakers on the panel emphasized that scholars studying these issues should be humble, sensitive, and do no harm, which is a message Irwin Feller delivered at the beginning of the workshop. The returns on federal investments in research are extremely complex and occur within the context of a complex economy and society. Analysts should avoid claiming more for the utility of their work than is warranted.