Excerpt

On April 18–19, 2011, the Board on Science, Technology, and Economic Policy (STEP) of the National Research Council and the Committee on Science, Engineering and Public Policy (COSEPUP), a joint unit of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine, held a workshop to examine this question The workshop brought together academic researchers, research and development (R and D) managers from private industry, representatives from government agencies, leaders of philanthropic organizations, and others to look at the very broad range of issues associated with evaluating the returns on federal investments (Appendix A). Speakers included researchers who have worked on the topic for decades and early-career researchers who are pioneering non-traditional approaches to the topic. In recent years, new research has appeared and new data sets have been created or are in development. Moreover, international interest in the topic has broadened substantially—in Latin America and Asia as well as in Europe. The workshop included presentations by speakers from abroad to gain their perspectives on methods of analysis. The workshop sought to assemble the range of work that has been done in measuring research outcomes and to provide a forum to discuss its methods. The workshop’s goal was not to identify a single best method or few best methods of measuring research impacts. The workshop considered methodological differences across fields of research to identify which can be applied to the broad range of federal research funding. It did not address the role of federal funding in the development of technology.

Contents

• The National Academies
• Committee on Measuring Economic and Other Returns on Federal Research Investments
• Board on Science, Technology, and Economic Policy
• Committee on Science, Engineering, and Public Policy
• Acknowledgement of Reviewers
• 1. INTRODUCTION AND OVERVIEW
• 2. THE USES AND MISUSES OF PERFORMANCE MEASURES
  • THE PROMISE AND THE LIMITS OF MEASURING THE IMPACT OF FEDERALLY SUPPORTED RESEARCH
  • INNOVATION AS AN ECOSYSTEM
  • OVERCOMING THE CHALLENGES OF RESEARCH MEASURES
  • DISCUSSION
• 3. IMPACTS ON THE U.S. ECONOMY AND QUALITY OF LIFE
  • FEDERAL RESEARCH AND PRODUCTIVITY
  • INDIRECT ECONOMIC BENEFITS OF RESEARCH
  • BEYOND CITATIONS AND PATENT REFERENCE COUNTS
  • DISCUSSION
• 4. IMPACTS ON BIOMEDICAL AND HEALTH RESEARCH
  • REVIEWING THE LITERATURE ON HEALTH IMPACTS
  • THE VOLATILITY OF FEDERAL R AND D SUPPORT
  • MEDICAL DEVICE INNOVATION
  • MAKING DECISIONS IN THE PHARMACEUTICAL INDUSTRY
  • RESEARCH AND OUTCOMES CASE STUDY: PEDIATRIC HIV
  • DISCUSSION
• 5. MIXED MARKET AND NON-MARKET IMPACTS OF RESEARCH
  • MEASURING PROGRESS TOWARD GOALS IN AGRICULTURAL PRODUCTIVITY
  • INVESTMENT DECISIONS AT DUPONT
  • CHALLENGES IN QUANTIFYING RESEARCH VALUE IN AGRICULTURE
  • MEASURING SUCCESS IN CONSERVATION
  • NATIONAL SECURITY BENEFITS
  • PUBLIC PROBLEM SOLVING
  • DISCUSSION
• 6. IMPACTS OF RESEARCH ON THE LABOR MARKET AND CAREER DEVELOPMENT
  • R AND D SPENDING AND THE R AND D WORKFORCE
  • SURVEYS OF GRADUATE STUDENTS AND POSTDOCTORAL FELLOWS
  • THE COMPLEX NETWORK OF SKILLS AND INVESTMENTS
  • DISCUSSION
• 7. INTERNATIONAL PERSPECTIVES ON MEASURING RESEARCH IMPACTS
  • MEDICAL RESEARCH COUNCIL EVALUATION SYSTEM
  • MEASURING IMPACTS OF RESEARCH FUNDING IN THE EUROPEAN UNION
  • MEASURING IMPACTS OF SCIENCE, TECHNOLOGY, AND INNOVATION INVESTMENTS IN BRAZIL
  • DISCUSSION
• 8. EMERGING METRICS AND MODELS
  • ASSESSING RESEARCH AT NSF
  • THE STAR METRICS PROJECT
  • RECONSTRUCTING NETWORKS OF DISCOVERY
  • CREATING KNOWLEDGE FROM DATA
  • MEASURING THE IMPACT OF STAR SCIENTISTS
  • VISUAL ANALYTICS
  • CONSIDERATIONS IN BUILDING COMPREHENSIVE DATABASES
  • DISCUSSION
• 9. PITFALLS, PROGRESS, AND OPPORTUNITIES
  • PITFALLS ON THE ROAD TO UNDERSTANDING
  • PROGRESS IN UNDERSTANDING THE ISSUES
  • OPPORTUNITIES POSED BY GREATER UNDERSTANDING
  • CONCLUDING REMARKS
• REFERENCES
• APPENDIX A WORKSHOP AGENDA
• APPENDIX B BIOGRAPHICAL INFORMATION
• APPENDIX C THE PROMISES AND LIMITATIONS OF PERFORMANCE MEASURES
• APPENDIX D THE IMPACT OF PUBLICLY FUNDED BIOMEDICAL AND HEALTH RESEARCH: A REVIEW

Rapporteurs: Steve Olson and Stephen Merrill

This study was supported by Contract/Grant No. SMA-1019816 between the National Academy of Sciences and the National Science Foundation; Contract/Grant No. N01-OD-4-2139, TO #231, between the National Academy of Sciences and the National Institutes of Health; Contract/Grant No. G104P00159 between the National Academy of Sciences and the U.S. Geological Survey; Contract/Grant No. 59-9000-0-0093 between the National Academy of Sciences and the U.S. Department of Agriculture; Contract/Grant No. EP-11-H-001414 between the National Academy of Sciences and the Environmental Protection Agency; Contract/Grant No. DE-SC000614 between the National Academy of Sciences and the Department of Energy; Contract/Grant No. NNH10CC488, TO #5, between the National Academy of Sciences and NASA.

Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.