The Study Charge

This study will review the SBIR program at the five agencies with regard to parameters such as the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program's contribution to accomplishing agency missions. To the extent possible, the evaluation will include estimates of the benefits (both economic and non-economic) achieved by the SBIR program. The study will also examine broader policy issues associated with public-private collaborations for technology development and government support for high technology innovation. The project will encourage cross-fertilization among program managers, agency officials, and participants by convening national experts from industry, academia, and the public sector to review and discuss research findings. Where appropriate, operational improvements to the program will be considered.

The Objectives of the Study

The objectives of the study are to:

• Satisfy the Congressional mandate for an objective, external assessment of the program;
• Provide an empirical analysis of the operations of the SBIR program, in particular rates and sources of commercialization, for agency officials and program managers;
• Address research questions relevant to the program's operation and evaluation derived from the legislation and that emerge in the course of the study;
• Develop a rigorous assessment of the program and contribute to Congressional understanding of its multiple objectives, measurement issues, operational challenges, and contributions as described in the legislation.

Focus of the Evaluation

Following the passage of HR 5667 in December 2000, extensive discussions were held between the NRC and the five leading agencies regarding the scope and nature required to fulfill the Congressional mandate. Agreement on the terms of the study was reached in December 2001, and the requisite funding for the Academies to begin the study was received in September 2002. The study was officially launched on 1 October 2002. The Memorandum of Understanding between the NRC and the agencies reflects the Congressional mandate by specifying a particular focus for the evaluation on four aspects of the SBIR Program:

1. Commercialization. Congress established the SBIR program partly to support commercialization of Federal research. The agencies have in general interpreted this to mean support for research activities, which could result in successful commercialization, measured in different ways, while also meeting other objectives.
2. Mission support. Congress has also mandated that SBIR programs should support the mission of the funding agency. Of course, each agency has a different mission, which means that, at least in part, different indicators or metrics will be needed. Indeed, initial research indicates that there are very significant differences in this area among agencies that fund high tech research in order to eventually purchase the outputs from goods and services that may emerge from it (DoD, NASA, parts of DoE) and those that do not (NSF, NIH, parts of DoE). This basic difference suggests the need for quite different research strategies, but, in both cases, the study seeks to establish the extent to which SBIR programs meet this component of the Congressional mandate.
3. Knowledge base. All federal research includes the objective of expanding the nation's knowledge base. SBIR programs are also charged with this objective, which appears to be doubly important for the non-procuring agencies. For these non-procuring agencies, a substantial part of the agency mission could also be described as to the expansion of the knowledge base, through intermediate and final products.
4. Program management. The charge to the Committee includes the provision of recommendations for improving the SBIR program (although not for assessing its continued existence). That charge requires review and assessment of how each agency SBIR program operates, and—where possible—the identification of best practices and possible improvements.

Limits of the Committee Charge

The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively on this question. Rather, the NRC Committee conducting this study is charged with providing assessment-based findings of the benefits and costs of SBIR (described in the Objectives section above) to improve public understanding of the program, as well as recommendations to improve the program's effectiveness. It is also important to note that, in accordance with the Memorandum of Understanding and the Congressional mandate, the study will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area.

A Two-Phase Study Structure

The project is divided into two phases. Phase I has focused on data collection and the development of the methodology. Per the agreement with the agencies as outlined in the Memorandum of Understanding,³ this Methodology was submitted to an intensive Academy review process, involving 12 reviewers with recognized expertise in economics, statistics, program evaluation, survey methodology, innovation policy, federal R&D programs, and both large and small high-tech firms.

Extensive revisions and elaborations were required as a result of this review and they are now reflected in this document. The second phase of the study will now implement the research methodology developed in Phase I. Phase I included an initial symposium for the program as a whole, followed by a number of committee meetings and a series of workshops to address the specific features of each agency's program. This phase has focused on the development of survey instruments, case study templates, and related research to the extent possible. Additional details regarding the study methodologies to be used have been deferred in some cases because they cannot be defined precisely until some initial Phase II work has been completed. At the conclusion of Phase I, the overall methodology and evaluation tools will be submitted for review. In Phase II, research papers on general topics will be commissioned, and preliminary results of field research will be assessed and cross-checked. A symposium will be convened to discuss publicly the results of the research, and final reports will be prepared for each agency and for the program as a whole.

Summary of Methods to be Used

The purpose of this document is to describe the methodological approaches developed under Phase I of the study. They build from the precedents established in several key studies already undertaken to evaluate various aspects of the SBIR. These studies have been successful because they identified the need for utilizing not just a single methodological approach, but rather a broad spectrum of approaches, in order to evaluate the SBIR from a number of different perspectives and criteria.

This diversity and flexibility in methodological approach are particularly appropriate given the heterogeneity of goals and procedures across the five agencies involved in the evaluation. Consequently, this document suggests a broad framework for methodological approaches that can serve to guide the research team when evaluating each particular agency in terms of the four criteria stated above. Table 1 illustrates some key assessment parameters and related measures to be considered in this study.⁴

TABLE 1

OVERVIEW OF APPROACH TO SBIR PROGRAM ASSESSMENT.

Footnotes

3

See Annex C in this volume.

4

See also Annex F in this volume.