I. NATIONAL RESEARCH COUNCIL (NRC) STUDY FINDINGS

1. The Small Business Innovation Research (SBIR) Program Is Making Significant Progress in Achieving the Congressional Goals for the Program. The SBIR program is sound in concept and effective in practice. With the programmatic changes recommended here, the SBIR program should be even more effective in achieving its legislative goals.
   1. Overall, the Program Has Made Significant Progress in Achieving its Congressional Objectives by:
      1. Stimulating Technical Innovation. By a variety of metrics, the program is contributing to the nation’s stock of new scientific and technical knowledge. (See Finding F.)
      2. Using Small Businesses to Meet Federal Research and Development Needs. SBIR program objectives are aligned with, and contribute significantly to fulfilling the mission of each studied agency.¹ In some cases, closer alignment and greater integration should be possible. (See Finding C).
      3. Fostering and Encouraging Participation by Minority and Disadvantaged Persons in Technological Innovation. The SBIR program supports the growth of a diverse array of small businesses, including minority- and woman-owned business, by providing market access, funding, and recognition. To better assess this support, enhanced efforts to collect better data and to monitor outcomes more closely are required at some agencies. Also, more analysis is needed to improve understanding of the obstacles faced by minority- and woman-owned businesses and the nature of the measures needed to address these obstacles. (See Findings D and E.)
      4. Increasing Private Sector Commercialization of Innovation Derived from Federal Research and Development. The program enables small businesses to contribute to the commercialization of the nation’s R&D investments, both through private commercial sales, as well as through government acquisition, thereby enhancing American health, welfare, and security through the introduction of new products and processes. (See Finding B.)
   2. SBIR Is Meeting Federal R&D Needs. SBIR plays an important role in introducing innovative, science-based solutions that address the diverse mission needs of the federal agencies.
      1. The program has the potential to make systematic investments in high-potential, high-risk technologies that can meet specific mission needs and/or offer the potential of significant commercial development. Balancing the potential gains from high risk, but potentially high payoff, against technologies with more promising immediate commercial benefits, represents a challenge for management and appropriate evaluation.
   3. Improving SBIR. As structured, the management of SBIR exhibits considerable flexibility, allowing program experimentation needed to address varied agency missions and technologies. This flexibility is commendable and has enabled the program to contribute to multiple agency missions. At the same time, more regular assessment is required, both internal and external, to inform agency management of program outcomes and to improve performance through analysis of program outcomes and regular adoption of agency “best practices.” (See Section II on Recommendations.)
2. Commercialization.
   1. Commercialization Despite High Risk. Small technology companies use SBIR awards to advance projects, develop firm-specific capabilities, and ultimately create and market new commercial products and services.
      1. Given the very early stage of SBIR investments, and the high degree of technical risk often involved (reflected in risk assessment scores developed during agency selection procedures), the fact that a high proportion of projects reach the market place in some form is significant, even impressive.
      2. Although the data vary by agency, responses to the NRC Phase II Survey indicate that just under half of the projects do reach the marketplace.²
   2. Multiple Indicators of Commercial Activity. The full extent of commercialization for both government and private use is insufficiently captured by simply recording sales of a product or service. The Committee’s analysis uses five measures of commercial activity, although sales in the marketplace are obviously of particular importance. These indicators are:
      1. Sales (using a range of different benchmarks to indicate different degrees of commercial activity)
      2. Additional non-SBIR research funding and contracts
      3. Licensing revenues
      4. Third-party investment (including both venture funding and other sources of investment)
      5. Additional SBIR awards for related work
   3. A Skewed Distribution of Sales. SBIR awards result in sales numbers that are highly skewed, with a small number of awards accounting for a very large share of the overall sales generated by the program.³ This is to be expected in funding early-stage technological innovation and is broadly consistent with the general experience of other sources of early technology financing by angel investors.⁴ Most projects, however, do not achieve significant commercial success; a few companies do.⁵
      1. As an example, just eight of the NSF-supported projects—each of which had $2.3 million or more in sales—accounted for over half of the total sales dollars reported by respondents.
      2. Similarly, one firm selected for the NRC Phase II Survey accounted for over half of all licensing income that the 162 respondents reported they earned from the NSF SBIR-supported projects.⁶
   4. SBIR Is an Input, Not a Panacea. SBIR can be a key input to encourage small business commercialization, but most major commercialization successes require substantial post-SBIR research and funding from a variety of sources. SBIR awards will have been in many cases a major, even critical input—but only one of many inputs.
   5. SBIR Projects Attract Significant Additional Funding. SBIR funded research projects enable small businesses to develop the technical know-how needed to attract third-party interest from a variety of public and private sources, including other federal R&D funds, angel investors, and venture funds. The NRC survey revealed that 56 percent of surveyed projects were successful in attracting additional funding from a variety of sources.⁷
      1. Federal Funding. Responses to the NRC Phase II Survey indicate that about 10 percent of Phase II projects were eventually supported by other federal research funding; over half received at least one ad ditional related Phase I SBIR award, and more than half received at least one additional related Phase II award;⁸
      2. Licensing. Sixteen percent of projects report licensing agreements in place for their technologies, with an additional 16 percent engaged in licensing negotiations at the time of the NRC Phase II Survey.⁹ Some companies are making substantial use of the technology.¹⁰
      3. Venture Funding. Venture capital (VC) funding has, in some cases, also played a significant role for SBIR award winners. At NIH, for example, some 25 percent of the 200 companies that received the highest number of SBIR awards had also obtained funding from venture capital, totaling some $1.59 billion.¹¹
      4. Acquisition. In some cases, the SBIR funded technology developed sufficient commercial potential that investors bought the grantee company outright. For example, in 2000, Philips bought out SBIR recipient Optiva, reportedly for the sum of $1 billion.¹²
   6. Better Documentation of Commercialization Is Needed. While the recorded successes of the program are significant, commercialization outcomes are not yet adequately documented in most of the agencies reviewed. Indeed, some agencies have made very limited efforts to track the evolution of firms having received SBIR awards. Understanding outcomes and the impact of program modifications on those outcomes is essential for effective program management.
3. Effective and Flexible Alignment of SBIR with Agency Mission.
   1. Flexible Program Management. The effective alignment of the program with widely varying mission objectives, needs, and modes of operation is a central challenge for an award program that involves a large number of departments and agencies. The SBIR program has been adapted effectively by the management of the individual departments, services, and agencies, albeit with significant operational differences, reflecting distinct missions and operational cultures. This flexibility in program management is one of the great strengths of the program.
   2. Mission Support. Each of the agency SBIR programs is effectively supporting the mission of the respective funding agency:¹³
      1. Defense.¹⁴ At DoD, considerable progress has been made in aligning SBIR-funded research with the strategic objectives of agency research and acquisition. In some parts of DoD, e.g., some Program Executive Officers within the Navy, significant success has been achieved in improving the insertion of SBIR-funded technologies into the acquisition process. The commitment of upper management to the effective operation of the program and the provisioning of additional funding appear to be a key element of success. Teaming among the agency, the SBIR awardees, and the prime contractors is important in the transition of technologies to products to integration in systems. The growing importance of the SBIR program within the defense acquisition system is reflected in the increasing interest of primes, who are seeking opportunities to be involved with SBIR projects—a key step toward acquisition.¹⁵
      2. NIH.¹⁶ NIH and other public health agencies increasingly see SBIR as an important element in the agency’s translational strategy—designed to move technologies from the lab into the marketplace.¹⁷ The NIH awards cases examined focused on public health and biomedical science and technology. The NRC’s review of the SBIR program at NIH documents that SBIR program funds projects have a significant impact on public health: NIH considers a project’s impact on public health during the selection process, and recipients and NIH staff note that examining potential impacts is an important component in every application review.¹⁸ The FDA’s Critical Path Initiative is focused on similar concerns.¹⁹
      3. NASA.²⁰ At NASA, the primary metric for project success is the deployment of SBIR-funded technologies on space missions, where the agency can point to a number of significant impacts.²¹
      4. Energy.²² At DoE, as at NASA, SBIR topics and project selection are heavily influenced by the agency research staff’s wider R&D responsibilities. At DoE, all technical topics in the SBIR solicitation are constructed to support the overall mission of each of the agency’s technical program areas.
      5. NSF.²³ At NSF, SBIR-funded research meets the agency’s primary mission of expanding scientific and technical knowledge.²⁴ However, the SBIR program does so by tapping the scientific and engineering capabilities of an important sector—small businesses—that is almost entirely unserved by the remainder of NSF’s programs.²⁵
4. The SBIR Program Provides Significant Support for Small Business.
   1. Substantial Benefits. The NRC Phase II Survey and NRC Firm Survey show that the SBIR program has provided substantial benefits for participating small businesses at all agencies, in a number of different ways. These benefits include:
      1. Encouraging Company Foundation. According to the NRC Firm Survey, just over 20 percent of companies indicated that they were founded entirely or partly because of an SBIR award;²⁶
      2. Supporting the Project Initiation Decision. Over two-thirds of SBIR projects reportedly would not have taken place without SBIR funding.²⁷ This finding reflects the known difficulties in funding commercial applications for early-stage technologies. The NRC Phase II Survey shows that SBIR seems to provide funding necessary to initiate early stage projects.
      3. Providing an Alternative Development Path. Companies often use SBIR to fund alternative development strategies, exploring technological options in parallel with other activities.
      4. Partnering and Networking. SBIR funding helps small businesses access outside resources, especially academic consultants and partners, helping to create networks and facilitating the transfer of university knowledge to the private sector.
      5. Commercializing Academic Research. In addition to fostering partnerships between academic institutions and private firms, SBIR plays an important role in encouraging academics to found new firms that can commercialize their research.²⁸
   2. SBIR Support Is Widely Distributed. SBIR provides support for innovation activity that is widely distributed across the small business research community.
      1. Large Number of Firms. During the fourteen years between 1992 and 2005, inclusive, more than 14,800 different firms received at least one Phase II award, according to the SBA Tech-Net database.²⁹
      2. Many New Participants. The agencies tracking new winners indicate that at least a third of awards at all agencies go to companies that had not previously won awards at that agency.³⁰ This steady infusion of new firms is a major strength of the program and suggests that SBIR is encouraging innovation across a broad spectrum of firms, creating additional competition among suppliers for the agencies with procurement responsibilities, and providing agencies with new mission-oriented research and solutions.
5. Benefits for Minority- and Woman-owned Small Businesses.
   1. A Key Program Objective. One of the four congressional objectives for the SBIR program is to enhance opportunities for woman- and minority-owned businesses. SBIR’s competitive awards provide a source of capital for small innovative firms with pre-prototype technologies—the phase where funding is most difficult to obtain. The certification effect that SBIR awards generate can be especially valuable as a signal to the early-stage capital markets of the potential of the project and hence of the firm.
   2. A Mixed Record. Woman- and minority-owned firms face substantial challenges in obtaining early-stage finance.³¹ Recognizing these challenges, the legislation calls for fostering and encouraging the participation of women and minorities in SBIR. Given this objective, some current trends are troubling. Agencies do not have a uniformly positive record in funding research by woman- and minority-owned firms. Efforts to document and monitor their participation have been inadequate at, for example, NIH. It is also true, however, that developing appropriate measures in this area is complex given, inter alia, the variations in the demographics of the applicant pool.³²
      1. Trends in Support for Woman-owned Businesses Vary Across Agencies.
         1. At DoD, which accounts for over half of the SBIR program funding, the share of Phase II awards going to woman-owned businesses increased from 8 percent at the time of the 1992 reauthorization (1992-1994) to 9.5 percent in the most recent years covered by the NRC Phase II Survey (1999-2001).³³ This percentage increase occurred in a program that expanded significantly over the same time period, both in terms of the R&D base and the SBIR allocation.
         2. At NSF, in recent years, woman-owned businesses have submitted an average of 213 Phase I proposals annually, and they have received an average of 26.6 Phase I awards annually. With the exception of the bump-up in 2002 and 2003, there is no upward trend. In other words, while woman-owned businesses do participate in the NSF SBIR program, that participation does not seem to be increasing over time.
         3. At NIH, awards to woman-owned businesses have increased, and their share of all awards is trending upward.³⁴ However, the percentage of female life scientists has been growing much faster.³⁵
      2. Support for Minority-owned Firms Has Not Increased Proportionately.
         1. DoD. The share of Phase I awards to minority-owned firms at DoD has declined quite substantially since the mid 1990s and fell below 10 percent for the first time in 2004 and 2005.³⁶ Data on Phase II awards suggest that the decline in Phase I award shares for minority-owned firms is reflected in Phase II.
         2. NSF minority firm participation is higher than that of woman-owned firms.³⁷ However, application rates have not been rising in line with changes in the demographics of the scientific and technical workforce, and success rates for woman- and minority-owned companies continue to lag those of other small businesses.³⁸
         3. NIH. Data only recently provided by NIH raises significant concerns about the shares of awards being made to minority-owned firms. The numbers of awards and applications, as well as success rates, have all declined for minorities, for both Phase I and Phase II. The lack of accurate and complete data suggests insufficient management attention to this component of the program’s objectives.³⁹
6. Stimulating Scientific and Technological Knowledge Advances. SBIR companies have generated many patents and publications, the traditional measures of activity in this area.
   1. Multiple Knowledge Outputs. SBIR projects yield a variety of knowledge outputs. These contributions to knowledge are embodied in data, scientific and engineering publications, patents and licenses of patents, presentations, analytical models, algorithms, new research equipment, reference samples, prototypes products and processes, spin-off companies, and new “human capital” (enhanced know-how, expertise, and sharing of knowledge).
   2. The NRC Phase II Survey found that 34 percent of NIH projects surveyed generated at least one patent, and just over half of NIH respondents published at least one peer-reviewed article.⁴⁰
   3. Linking Universities to the Public and Private Markets. The SBIR program supports the transfer of research into the marketplace, as well as the general expansion of scientific and technical knowledge, through a wide variety of mechanisms. With regard to SBIR’s role in linking universities to the market, about a third of all NRC Phase II and Firm Survey respondents indicated that there had been involvement by university faculty, graduate students, and/or a university itself in developed technologies. This involvement took a number of forms.⁴¹ Among the responding companies—
      1. More than two-thirds had at least one academic founder, and more than a quarter had more than one;
      2. About one-third of founders were most recently employed in an academic environment before founding the new company;
      3. In some 27 percent of projects, university faculty were involved as principal investigators or consultants on the project; d. 17 percent of Phase II projects involved universities as subcontractors; and
      4. 15 percent of Phase II projects employed graduate students.
      These data underscore the significant level of involvement by universities in the program and highlight the program’s contribution to the transition of university research to the marketplace.
   4. High-risk Focus. Projects funded by SBIR often involve high technical risk, implying novel and difficult research rather than incremental change. At NSF, for example, just more than half of the Phase I projects did not get a follow-on Phase II. One third of the Phase I awardees did not apply for Phase II awards, and of those that did, many did not get a follow-on Phase II for technical reasons. The chief reason for discontinuing Phase II projects was technical failure or difficulties.⁴² This effort to push the technological frontier is a strength of the program and necessarily involves project, but not program, failure.
   5. Indirect Path Effects. There is strong anecdotal evidence concerning “indirect path” effects—that investigators and research staff gain knowledge from projects that may become relevant in a different technical context later on, and often in another project or even another company. These effects are not directly measurable, but interviews and case studies affirm their existence and importance.⁴³
7. The Measurement Challenge. The diversity of missions among agencies involved with SBIR means that metrics and indicators that capture common goals and others that reflect the diversity in mission and mechanisms (e.g., contracts or grants) must be employed to measure outcomes and impacts.
   1. Differing Goals. A fundamental difference exists between those agencies that procure SBIR-funded products for their missions and those that normally do not. At DoD, the program is driven by the need to put the results of research into products and systems to support war-fighters; at NIH, the focus of the SBIR program is to conduct research that contributes to improvements in public health, almost always through eventual take-up by entities outside NIH.⁴⁴
   2. Differing Mechanisms. These distinctly different goals are reflected in the mechanisms used by the agency to implement the SBIR program.⁴⁵ For example, the DoD employs contracts whereas the NIH generally employs grants for its SBIR awards.
   3. Different Metrics. Developing metrics that can capture impacts in both of these areas is a challenge. Measuring the extent to which the program supports the agency mission in any nonprocurement agency like NSF is substantially different from measures applicable to a procurement agency like DoD or NASA. In some cases, the products of NIH-funded research need to be taken up by the private sector in order to be further developed before their contribution can be realized. In other cases, such as diagnostics, software, and educational materials to encourage better health, the SBIR awards can be sufficient in themselves.
8. SBIR Program Flexibility Is a Positive Feature and Should Be Preserved. As structured, the management of SBIR exhibits commendable flexibility. SBA and agency management have allowed the program managers the room to adapt the program to the needs of specific technologies and unique mission needs.
   1. Multiple Missions. The varied missions of the agencies that fund SBIR have encouraged SBA to take a flexible approach to implementing program guidelines. SBA has provided waivers on funding size and other programmatic matters, and agencies have used considerable ingenuity in exploring the boundaries of the guidelines, particularly in terms of support for commercialization.⁴⁶
   2. Commendable Flexibility. This flexibility is highly commendable and should be expanded where possible. It is precisely the relatively wide range of options available to the agencies and their SBIR program managers that helps to account for the effectiveness of the program. The need for flexibility is grounded in the very wide differences in objectives, structure, and culture across the agency SBIR programs as well as in the uncertainties of early stage project finance.⁴⁷
   3. Program Diversity.
      1. Mission Agencies. As noted above, agencies with substantial hardware and systems procurement responsibilities rely primarily on contracts as the vehicle for commissioning research—including SBIR. Contracts are oriented toward the production of results for which the contracting agency has a specific need, often via a contract with strict terms and conditions, and clear deliverables. In a contracts environment, Phase I is seen as a means of testing the feasibility of solutions for a problem, and a single Phase II may be selected to implement the most promising of these options.
      2. Research Agencies. Agencies without significant hardware or systems procurement generally use grants. Grants awarded to support promising academic proposals are often initiated by researchers with interests in a specific area. At NIH, for example, topics in the SBIR solicitation serve only as guidance to areas of interest, underscoring the NIH commitment to investigator-driven research. For research agencies, almost all sales are to the private sector, and the sequential funding approach of DoD is replaced by a wider competition among all Phase I and Phase II proposals in a given technical area.
      3. Consequences of Different Approaches. Contracts and grants carry with them important differences in approach and agency mindset, and consequently in outcomes, metrics, and evaluation benchmarks. Such differences permeate all aspects of the program, and underscore the need for a very flexible approach to program guidelines. The grants-oriented approach at NIH has included substantial flexibility in the application of deadlines for completion, award size, and supplemental funding among other program attributes. The contracts-driven approach at DoD has resulted in tight deadlines, more limited funding flexibility within the program, and program goals based on clear deliverables.
   4. Commendable Innovation in Program Operation. Since the late 1990s, the SBIR program has seen considerable innovation in its operations, a trend that has accelerated during this study. As the list below shows, some SBIR program managers have increasingly utilized the flexibility inherent in the structure of such a widely dispersed program to introduce a range of initiatives covering almost all aspects of SBIR program management.
      1. These include innovations and initiatives related to:
         • Award size and duration.
         • Topic development.
         • Gap funding.
         • Phase III commercialization support.
         • Matching funds for commercialization.
         • Commercialization training for SBIR awardees.
         • Outreach into the small business community, especially to underrepresented states.
         • Increased evaluation and assessment activity.
      2. While not all of these changes have been made at each agency, these changes taken as a whole do indicate that the program is increasingly benefiting from an innovative management culture and is becoming better adapted to serve its goals. These changes also suggest that there are likely to be further innovations and adaptations of the program to the changing needs of agencies’ missions.
   5. Procedural Issues. A review of topic development and selection procedures, which vary among agencies, indicated that in general, agencies work hard to ensure that applicants are treated fairly, and that topics are developed in line with agency needs. Specific recommendations for improving program management are provided in subsequent sections of this report.
   6. The Need for Regular Internal and External Evaluation. While the initiatives taken by several agencies to adapt their program to serve their mission needs more effectively are welcome and their flexibility to do so is commendable, these efforts require periodic assessment to determine if these ‘operational experiments’ are successful and to identify the appropriate lessons learned. Moreover, some important innovations do not appear to be based on any detailed evaluation of problems with the existing mechanisms.
   7. SBA and SBIR. The SBA has oversight responsibility for the eleven SBIR programs underway across the federal government. The agency is to be commended for its flexibility in exercising its oversight responsibilities. As noted above, the agencies adaptation of the program to fit their needs and methods of operation has proven fundamental to the program’s success.
      1. Modest Resources. In general, SBA has been allocated modest resources to oversee SBIR. For example, efforts to build a unified database continue to encounter difficulties and resources are apparently a constraint.
      2. More Flexibility. Despite the generally flexible approach, there is evidence from the agencies that in some cases, SBA has believed itself obliged to limit flexibility in the application of its guidelines to the program. For example, projects initially funded as a Phase I STTR cannot be shifted to a Phase II SBIR, even if circumstances make this the most effective way to proceed.⁴⁸
9. Venture Funding and SBIR.
   1. Synergies. There can often be useful synergies between angel and venture capital investments and SBIR funding; each of these funding sources tends to select highly promising companies and technologies. Moreover, an SBIR award provides information to potential investors by validating the technology concept and commercial potential of an innovation.
      1. Angel Investment. Angel investors often find SBIR awards to be an effective mechanism to bring a company forward in its development to the point where risk is sufficiently diminished to justify investment.⁴⁹
      2. Venture Investment. Reflecting this synergy, initial NRC review indicates about 25 percent of the top 200 NIH Phase II award winners (1992-2005) have acquired some venture funding in addition to the SBIR awards.⁵⁰
   2. Program Change. During the first two decades of the program, some venture-backed companies participated in the program, receiving SBIR awards in conjunction with outside equity investments. During this lengthy period, no negative impact on the program’s operation or effectiveness was apparent.
      In a 2002 directive, the Small Business Administration said that to be eligible for SBIR the small business concern should be “at least 51 percent owned and controlled by one or more individuals who are citizens of, or permanent resident aliens in, the United States, except in the case of a joint venture, where each entity to the venture must be 51 percent owned and controlled by one or more individuals who are citizens of, or permanent resident aliens in, the United States.”⁵¹ The effect of this directive has been to exclude companies in which VC firms have a controlling interest.^52,53
      1. It is important to keep in mind that the innovation process often does not follow a crisp, linear path. Venture capital funds normally (but not always) seek to invest when a firm is sufficiently developed in terms of products to offer an attractive risk-reward ratio.⁵⁴ Yet even firms benefiting from venture funding may well seek SBIR awards as a means of exploring a new concept, or simply as a means of capitalizing on existing research expertise and facilities to address a health-related need or, as one participant firm explained, to explore product-oriented processes not “amenable to review” by academics who review the NIH RO1 grants.⁵⁵
      2. Some of the most successful NIH SBIR award winning firms—such as Martek—have, according to senior management, been successful only because they were able to attract substantial amounts of venture funding as well as SBIR awards.⁵⁶
      3. Other participants in the program believe that companies benefiting from venture capital ownership are essentially not small businesses and should therefore not be entitled to access the small percentage of funds set aside for small businesses, i.e., the SBIR Program. They believe further that including venture-backed firms would decrease support for high-risk innovative research in favor of low-risk product development often favored by venture funds.⁵⁷
   3. Limits on Venture Funding. The ultimate impact of the 2004 SBA ruling remains uncertain. What is certain is that no empirical assessment of its impact was made before the ruling was implemented. At the same time, the claims made by proponents and opponents of the change appear overstated.
      1. Preliminary research indicates that approximately 25 percent of the most prolific NIH SBIR Phase II winners have received VC funding; that some of these are now graduates of the program (having grown too large or left for other reasons), and some are also not excluded by the ruling because they are still less than 50 percent VC owned. Yet it is important to recognize that these companies may be disproportionately among the companies most likely to succeed—such as previous highly successful SBIR companies that were simultaneously recipients of VC funding.⁵⁸ What is not known is how many companies are excluding themselves from the SBIR program as a result of the ruling.
      2. For firms seeking to capitalize on the progress made with SBIR awards, venture funding may be the only plausible source of funding at the levels required to take a product into the commercial marketplace. Neither SBIR nor other programs at NIH are available to provide the average of $8 million per deal currently characterizing venture funding agreements.⁵⁹
      3. For firms with venture funding, SBIR may allow the pursuit of high-risk research or alternative path development that is not in the primary commercialization path, and hence is not budgeted for within the primary development path of the company.⁶⁰
   4. An Empirical Assessment. As noted above, the SBA ruling concerning eligibility alters the way the program has operated during the period of this review (1992-2002) and, presumably, from the program’s origin. Anecdotal evidence and initial analysis indicate that a limited number of venture-backed companies have been participating in the program. To better understand the impact of the SBA exclusion of firms receiving venture funding (resulting in majority ownership), the NIH recently commissioned an empirical analysis by the National Academies. This is a further positive step towards an assessment culture and should provide data necessary to illuminate the ramifications of this ruling.⁶¹

II. NRC STUDY RECOMMENDATIONS

The recommendations in this section are designed to improve the operation of the nation’s SBIR program. They complement the core findings that the program is largely addressing its legislative goals—that significant commercialization is occurring, that the awards are making valuable additions to nation’s stock of scientific and technical knowledge, and that SBIR is developing products that apply this knowledge to agency missions, and hence to the nation’s needs.

1. Preserve Program Flexibility. Agencies, SBA, and the Congress should seek to ensure that any program adjustments made should not reduce the program’s flexibility.
   1. The SBIR program is effective across the agencies partly because a “one-size-fits-all” approach has not been imposed.
   2. This flexible approach should be continued, subject to appropriate monitoring, across the departments and agencies in order to adapt the program to their evolving needs and to improve its operation and output.
2. Regular Evaluations Are Needed.
   1. The SBIR Program Is Currently Not Sufficiently Evidence-based. Some aspects of the SBIR program have been subject to internal and external evaluation.⁶² Nonetheless, insufficient data collection, analytic capability, and reporting requirements, together with the decentralized character of the program, mean that there is limited ability to make connections between program outcomes and program management and practices. There are several issues. These include:
      1. Limited Collection of Data and Tracking of Outcomes. There is insufficient tracking of awards, as well as a widespread lack of systemic data collection and analysis, needed to support an evidence-based program culture and to enable program managers to be more effective in their stewardship of the program.
      2. Limited Analyses and Use of Metrics. Changes to the program need to be better understood and the impact tracked. Because of the factors listed above, decisions that affect the operation of the program are necessarily taken with limited data and analysis, and without clear benchmarks or metrics for assessing the success or failure of a given initiative.⁶³
      3. Inadequate Management Funding. Agencies administering SBIR can improve the management of the program by providing greater support for data collection and analysis. The SBIR program involves a significant amount of public funding, some $1.851 billion in FY2005. The necessary resources to manage the program and evaluate outcomes should be provided.⁶⁴
   2. Single Benchmarks of Achievement Are Problematic. The current study represents a major research effort in an area with limited previous research. Differences in both program structures and agency objectives mean that no simple benchmark for success is applicable across the entire program. Existing studies, such as those by GAO, have generally been limited in scope, focusing in most cases only on a limited aspect of the program.⁶⁵ This is not surprising, given the size and diversity of the program and the difficulties in developing the comprehensive data required for accurate assessments of program outcomes (described above). To address this assessment challenge, regular, multidimensional, multiagency evaluation is required.
   3. Developing a Culture of Evaluation. Recent agency initiatives toward more and better evaluation, e.g., at DoD, NIH, and NSF, represent positive steps. However, considerable further assessment is needed. The lack of an adequate assessment effort is, in some cases, the result of shortfalls in the resources needed to carry out assessments, and in others because agencies do not have the support of agency leadership for a sustained effort of evaluation and assessment.
   4. Recommendations. More evaluations of the SBIR program’s outcomes are needed. Such assessment can provide the data needed to inform program management decisions and to enhance understanding of the program’s contributions.
      1. Annual Reports. Top agency management should make a direct annual report to Congress on the state of the SBIR program at their agency. The report should describe management goals and commitment to the program, and measures taken to enhance the implementation of the program. This report should include a statistical appendix, which would provide data on awards, processes, outcomes, and survey information.
      2. Internal Evaluation. Agencies should be encouraged—and funded—to develop improved data collection technologies and evaluation procedures. Where possible, agencies should be encouraged to develop interoperable standards for data collection and dissemination.⁶⁶
      3. External Evaluation. Agencies should be directed to commission an external evaluation of their SBIR programs on a regular basis.
      4. Special Topic Studies. A number of further studies on special topics might help to improve program outcomes. These include:
         1. The role of venture capital funding in the SBIR program.
         2. The linkages and synergies between state programs and SBIR.
         3. Improving links between SBIR and the acquisition community at DoD: Effects of matching fund programs.
         4. Tools for reducing cycle time.
         5. Gap funding mechanisms and results.
         6. Phase III transition incentives.
         7. Impact of increased award size.
         8. Assessment of selection mechanisms.
         9. Evaluation of professional commercialization consultancy services.
3. Additional Management Resources Are Needed.
   1. To enhance program utilization, management, and evaluation, the program should be provided with additional funding for management and evaluation.
   2. Effective management and evaluation requires adequate funding.⁶⁷ An evidence-based program requires high quality data and systematic assessment. As noted above, sufficient resources are not currently available for these functions.
   3. Increased funding is needed to provide effective oversight, including site visits, program review, systematic third-party assessments, and other necessary management activities.
   4. In considering how to provide additional funds for management and evaluation, there are three ways that this might be done:
      1. Additional funds might be allocated internally, within the existing budgets of the services and agencies, as the Navy has done.
      2. Funds might be drawn from the existing set-aside for the program to carry out these activities.
      3. The set-aside for the program, currently at 2.5 percent of external research budgets, might be marginally increased, with the goal of providing management resources necessary to maximize the program’s return to the nation.⁶⁸ At the same time, increased resources should not be used to create new or separate management. Increased resources should go to enable existing management to enhance their efforts with respect to the program. The Committee recommends this third option.
4. Improve Program Processes.
   1. The processing periods for awards vary substantially by agency, and appear to have significant effects on recipient companies.⁶⁹
      1. Cycle time is a concern for small high-technology businesses with limited resources. Such small companies often have a substantial “burn rate” (outgoing monthly cash flow), and the speed with which projects are approved and funded can have a material impact on small business ventures.
      2. Numerous entrepreneurs interviewed for this study identified long cycle time—that is between initial application and Phase I awards, and between Phases—as a significant problem. Some case study interviewees noted that these gaps discourage otherwise excellent companies from applying to the SBIR program.⁷⁰
      3. Processing schedules that are adequate for managing larger-scale research projects or academic research may not be appropriate for the small, high-technology firms that apply for SBIR awards.
   2. Agencies Should Be Encouraged to Employ a Full Range of Tools to Reduce the Time Between Applications and Awards.
      1. Monitoring and Reporting. As part of a new and expanded annual report, agencies should closely monitor and report on cycle times for each element of the SBIR program: topic development and publication, solicitation, application review, contracting, Phase II application and selection, and Phase III contracting. Agencies should also specifically report on initiatives to shorten decision cycles.
      2. Best Practice Adoption. Agencies should seek to ensure that they adopt best practice in all areas of the program. Specifically, agencies should consider agency procedures that the Committee has identified that are expected to shorten the time from idea to implementation. Below we have provided a list of potential initiatives. Not all of these options would be appropriate for every agency, and agencies should be encouraged to experiment and assess the impact of such changes. Identified practices include:
         1. Multiple annual solicitations.
         2. Broader topic definitions.
         3. Solicitations where no topic is necessary, or where some funding is available to projects outside the topic framework.
         4. Rapid review and decisions on applications.
         5. Clear and tight deadlines for completing reviews.
         6. Permitting resubmission, and providing rejected applicants with reviews fast enough to ensure that they can apply again during the next cycle.
         7. Use of electronic tools to augment or replace traditional selection meetings.
         8. Rapid electronic delivery of results to applicants.
         9. Clear and tight deadlines for contracts and grants negotiations to be completed. This may require more resources and better training for awards management staff.
         10. Alignment of Phase I/Phase II applications, so that completion of Phase I comes at an appropriate time for Phase II application.
         11. Fast Track mechanisms such as that operating at NIH, where Phase I and Phase II are considered as a single application.
         12. Phase IIB awards which add additional agency resources on the condition that equal, or larger, amounts of private funding are also obtained.
         13. Further training of technical points of contact (TPOCs) and contracting officer’s technical representatives (COTARs) to help ensure that they understand the importance of shortening cycle times.
5. Agencies Should Take Steps to Increase the Participation and Success Rates of Woman- and Minority-owned Firms in the SBIR Program.
   1. Encourage Participation. Develop targeted outreach to improve the participation rates of woman- and minority-owned firms, and strategies to improve their success rates. These outreach efforts and other strategies should be based on causal factors determined by analysis of past proposals and feedback from the affected groups.⁷¹
   2. Encourage Emerging Talent. The number of women and, to a lesser extent, minorities graduating with advanced scientific and engineering degrees has been increasing significantly over the past decade, especially in the biomedical sciences. This means that many of the woman and minority scientists and engineers with the advanced degrees usually necessary to compete effectively in the SBIR program are relatively young and may not yet have arrived at the point in their careers where they own their own companies. However, they may well be ready to serve as principal investigators (PIs) and/or senior co-investigators (Co-Is) on SBIR projects. Over time, this talent pool could become a promising source of SBIR participants.
   3. Improve Data Collection and Analysis. The Committee also strongly encourages the agencies to gather the data that would track woman and minority firms as well as principal investigators (PIs), and to ensure that SBIR is an effective road to opportunity. The success rates of woman and minority PIs and Co-Is is the traditional measure of their participation in the non-SBIR research grants funded by agencies like NIH and NSF. It can also be a measure of woman and minority participation in the SBIR program.
6. Agencies that Do Not Now Have an Independent Advisory Board that Draws Together Senior Agency Management, SBIR Managers, and Other Stakeholders as Well as Outside Experts Should Consider Creating a Board to Review Current Operations and Achievements and Recommend Changes to the SBIR Program.
   1. The purpose of such an advisory board is to provide a regular monitoring and feedback mechanism that would address the need for upper management attention, and encourage internal evaluation and regular assessment of progress towards definable metrics.
   2. Each agency’s annual program report could be presented to the board. The board would review the report that would include updates on program progress, management practices, and make recommendations to senior agency officials in charge.
   3. The board could be assembled on the model of the Defense Science Board (DSB) or perhaps the National Science Foundation’s Advisory Board.⁷² In any case, it should include senior agency staff and the Director’s Office on an ex officio basis, and bring together, inter alia, representatives from industry (including award recipients), academics, and other experts in program management.
7. Preserve the Basic Program Structure.
   1. The Phase I Bypass Proposal. Some agency staff and recipient companies have suggested that promising research has been excluded from Phase II funding because all Phase II recipients must first receive a Phase I award.⁷³ To address this concern, some program participants and agency staff have suggested that consideration be given to changing the requirement that SBIR recipients apply for and receive a Phase I award before applying for Phase II. They suggest that the application of this requirement excludes promising research that could help agencies meet their congressionally mandated goals.
   2. Recommendation. The Committee recommends that fundamental changes to the Phase I, Phase II program structure should not be made.
      1. Permitting companies to apply directly to Phase II could significantly change the program. In particular, it could shift the balance of both awards and funding significantly away from Phase I toward Phase II.
      2. Every additional Phase II award represents funding approximately equivalent to 7.5 Phase I awards. If “direct to Phase II” were as attractive to applicants as proponents suggest, it might become a significant component of the program. This, in turn, could make a very substantial difference to funding patterns in SBIR, to the detriment of Phase I.⁷⁴
      3. This change could be detrimental to the program and should not be undertaken. Phase I is an important component in achieving congressional objectives and deemphasizing it should—at a minimum—require prior review and assessment.
8. Encourage Program Experimentation.
   1. The Fast Track Program, whose principal objective is to reduce the funding gap between Phase I and Phase II of the SBIR award process, began as an experiment at the Department of Defense. It was subsequently evaluated by the NRC, which found that the Fast Track Program increases the effectiveness of the SBIR program at DoD by encouraging the commercialization of new technologies. That report, published in 2000, recommended that DoD consider expanding the Fast Track Program within appropriate services, organizations, and agencies within DoD. That report also called for cross-agency comparisons of the impact of Fast Track, noting that it could prove useful for the continued refinement of the program.⁷⁵ DoD has since commissioned the NRC to conduct a follow up study of the Fast Track Program.
   2. Funding Beyond Phase II. SBIR is designed with two funded Phases (I and II) and a third nonfunded Phase III. Projects successfully completing Phase II are expected to be well placed to obtain Phase III funding from non-SBIR sources, e.g., through procurement contracts, to attract private investment, or to deploy products directly into the marketplace.
      1. While this three-phase approach has often proved successful, there are widely recognized difficulties in obtaining funding for Phase III, and consequently in achieving external commercial success or agency take-up in the agencies with major procurement responsibilities.⁷⁶
      2. Some agencies have sought, with the approval of SBA, to experiment with SBIR funding beyond Phase II in order to improve the commercialization potential of SBIR-funded technologies. NIH is now experimenting with tools for supporting projects through the early stages of regulatory review.
      3. The NSF Phase IIB initiative and the NIH Competing Continuation Awards are positive examples that might well be adapted elsewhere.
   3. Other Improvements. The agencies should be encouraged to develop program modifications to address possible improvements to the SBIR program, including but not limited to the pilot projects suggested above. (See Box 2-1.) SBA should make every effort to accommodate agency initiatives.
   4. Evaluation of Change. Agencies should equally ensure that program modifications are designed, monitored, and evaluated, so that positive and negative results can be effectively evaluated. Where feasible and appropriate, the agencies should conduct scientifically rigorous experiments to evaluate their most promising SBIR approaches—experiments in which SBIR program applicants, awardees, and/or research topics are randomly assigned to the new approach or to a control group that participates in the agency’s usual SBIR process.⁷⁷
9. Readjust Award Sizes.
   1. Erosion of Award Value.
      1. The real value of SBIR awards, last increased in 1995, has eroded due to inflation. It is now 14 years since the Congress increased the standard limits on the size of Phase I and Phase II awards. Many agency staff and award recipients have noted that in the face of continuing low but steady inflation, the amount of research funded by these awards has declined. Calculated using the NIH BioMedical Research and Development Price Index, the real value of the awards has declined by just over 35 percent.⁷⁸
      2. Given that Congress did not indicate that the real value of awards should be allowed to decline, this erosion in the value of awards needs to of awards needs to be addressed. However, many recipients contacted over the course of this study expressed concern that increases in the size of award would lead to a corresponding decline in the number of awards, if the SBIR budget is held constant.⁷⁹
      3. Steady increases in the number of applications for SBIR awards at most agencies (NIH is a notable recent exception) may be evidence that declining real award size has not led to a corresponding decline in interest among potential applicants.
   2. Recommendations.
      1. Phase I Increase. In order to restore the program to the approximate initial levels, adjusted for inflation, the Congress should consider making a one-time adjustment that would give the agencies latitude to increase the guidance on standard Phase I awards to $150,000.
      2. Phase II Increase. In order to restore the program to the approximate initial levels, adjusted for inflation and able to sustain quality applications, the Congress should consider making a one-time adjustment that would give the agencies latitude to increase the guidance on standard Phase II awards to approximately $1,000,000.⁸⁰
      3. Flexibility in Size of Awards. It should be stressed that recommendations are intended as guidance for standard award size. The SBA should continue to provide the maximum flexibility possible with regard to award size and the agencies should continue to exercise their judgment in applying the program standard. Recognizing agencies’ need for flexibility to meet new technical or mission challenges expeditiously—such as countermeasures for biological threats or Improvised Explosive Devices—strict limits on the minimum or maximum amount for awards should be avoided. The agencies, as well, should consider whether pilot programs offering larger (or indeed smaller) awards might be useful in some cases, and whether close evaluation of large awards made in the past could help guide future practice.⁸¹
      4. Flexibility—Duration. Contracting agencies might wish to experiment with a limited increase in flexibility, especially for Phase I where radical changes in technical direction are more likely to be required. Innovation implies uncertainty, so a more flexible timeline might help to attract approaches that are more technically innovative.
      5. Flexibility in Additional Funding. Agencies might consider providing supplementary awards to small businesses with promising technologies at the discretion of the program manager. Supplementary awards are made at NIH, and the NSF has, for some time, implemented a Phase IIB and Phase IIB+ program. The results of these initiatives should be assessed, not least because the results may be of significant value to other agencies.⁸² This de facto focus on the Phase III transition is especially important to those agencies that generally do not acquire the products of the firms receiving their awards.
      6. Transition. Larger award sizes should be phased in over two to three years in order to avoid a sudden reduction in the number of awards. The impact of the larger awards may be mitigated by the larger R&D budgets currently under consideration in the Congress. As suggested above, the transition in award sizes should be left to the individual agencies to implement in light of their current needs, capabilities, and priorities.
10. Understanding and Managing Firms Winning Multiple Awards.
    1. Unproductive Multiple Award Winners Do Not Appear to Be a Major Problem, Although Continued Monitoring Is Necessary.
       1. The common perception about the prevalence of mills in the SBIR program—i.e., that they have captured a large percentage of the awards, that they rarely commercialize, and that they do not meet agency research needs—is not substantiated by the evidence. Nonetheless, belief that the phenomenon of unproductive “mills,” that win repeated awards, is widespread. This perception has led to calls for a limit on the number of awards that a single firm can win.
       2. The NRC data suggest that firms that repeatedly win SBIR awards but fail to commercialize are not a significant problem at any of the granting agencies.⁸³ As the list of the top five Phase II winners (those with projects with $10 million or more in sales or investment) found in Table 2-1 further illustrates, most companies with multiple awards (but not all) are actually producing significant commercial products, often with substantial sales.
    2. Characteristics of Multiple Award Winners. Companies that do receive large numbers of awards at the contracting agencies (notably DoD) share two characteristics:
       1. If the award process is relatively objective, then one can argue that multiple award winners are high performers, able to address and meet agency needs, just as leading universities and major defense companies repeatedly win contracts for work based on a proven track record.
       2. As they grow in size, the dependence of frequent award winners on SBIR declines. They typically cease to fit the SBIR-dependent model of the mills outlined above.
    3. Recommendations. Because there is little evidence to suggest that multiple award winners are, in themselves, a problem, efforts to limit their participation seem misplaced. Limiting participation by unproductive winners is, of course, appropriate and already occurs. Setting a necessarily arbitrary limit on the number of awards seems neither necessary nor desirable in light of the contributions made by these firms, nor is it likely to be effective as firms could easily change names or spin out subsidiaries.
       1. Improved Monitoring. This phenomenon, however, should be monitored. As agencies improve their evaluation capacity, one area of focus could be multiple award winners. DoD, for example, should continue its efforts to track both awards to multiple winners and outcomes to ensure that they generate sufficient value to the defense mission, recognizing the varied contributions made by program participants. Better information on numbers of awards and especially on outcomes would, of course, enable management to make judgments that are more informed.
       2. The Use of Quotas to Reduce Applications by Multiple Winners Should Be Discontinued.
          1. In the case of multiple award winners who qualify in terms of the selection criteria, the acceptance/rejection decision should be based on both the strength of their applications and their past performance rather than on the number of grants received or applications made. Firms able to provide quality solutions to solicitations should not be excluded, a priori, from the program except on clear and transparent criteria (e.g., quality of research and/or past performance).
          2. Agencies should avoid imposing quotas unless there are compelling reasons to do so. Arbitrary limitations run the risk of limiting innovative ideas and of unnecessarily restricting opportunities among prospective principal investigators in the larger, eligible small companies to provide high quality solutions to the government.⁸⁴

BOX 2-1

Some Examples of Agency Best Practices. A major strength of the SBIR program is its flexible adaptation to the diverse objectives, operations, and management practices at the different agencies. In some cases, however, there are examples of best practice (more...)

TABLE 2-1

Top Five Phase II Winners: Projects with $10 Million or More in Sales or Investment.

III. SUMMARY: AN EFFECTIVE PROGRAM

In summary, the program is proving effective in meeting congressional objectives. It is increasing innovation, encouraging participation by small companies in federal R&D, providing support for small firms owned by minorities and women, and resolving research questions for mission agencies. Should the Congress wish to provide additional funds for the program in support of these objectives, with the programmatic changes recommended above, those funds could be employed effectively by the nation’s SBIR program.

Footnotes

1

The Department of Defense (DoD), National Institutes of Health (NIH), Department of Energy (DoE), National Aeronautics and Space Administration (NASA), and National Science Foundation (NSF).

2

See Figure 4-1. In addition, the response to the survey may reflect a degree of self-selection bias, e.g., the successful companies (those still in existence) are more likely to respond. Survey bias can manifest itself in several ways, however, and in some cases understate returns. For a fuller discussion of this important caveat, see Box 4-1 in Chapter 4: SBIR Program Outputs.

3

See Figure 4-2.

4

Even venture capital—a proximate referent group not appropriate for direct comparison because venture investments do not provide the same kind of project financing—shows a high returns skew. Nonetheless, returns on venture funding tend to show a similar high skew as characterize commercial returns on the SBIR awards. See John H. Cochrane, “The Risk and Return of Venture Capital,” Journal of Financial Economics, 75(1):3-52, 2005. Drawing on the VentureOne database, Cochrane plots a histogram of net venture capital returns on investments that “shows an extraordinary skewness of returns. Most returns are modest, but there is a long right tail of extraordinary good returns. 15 percent of the firms that go public or are acquired give a return greater than 1,000 percent! It is also interesting how many modest returns there are. About 15 percent of returns are less than 0, and 35 percent are less than 100 percent. An IPO or acquisition is not a guarantee of a huge return. In fact, the modal or ‘most probable’ outcome is about a 25 percent return.” See also Paul A. Gompers and Josh Lerner, “Risk and Reward in Private Equity Investments: The Challenge of Performance Assessment,” Journal of Private Equity, 1(Winter 1977):5-12. Steven D. Carden and Olive Darragh, “A Halo for Angel Investors,” The McKinsey Quarterly, 1, 2004, also show a similar skew in the distribution of returns for venture capital portfolios.

5

This is expected with research that pushes the state-of-the-art where technical failures are expected and when companies face the vagaries of the procurement process. A small proportion (3-4 percent) of projects generate more than $5 million in cumulative revenues. At NIH for example, only 6 of 450 projects in the recent NIH survey were identified as generating more than $5 million in revenues.

6

See National Research Council, An Assessment of the SBIR Program at the National Science Foundation, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2008, p. 214.

7

See NRC Phase II Survey, Questions 22 and 23, in Appendix A. SBIR awards are related to a range of other funding events for recipients, and interviews strongly suggest that SBIR does provide a positive validation effect for third party funders. This perspective is confirmed by some angel investors who see SBIR awards as a valuable step in providing funds to develop proof of concept and encourage subsequent angel investment. Steve Weiss, Personal Communication, December 12, 2006. Weiss, an angel investor, cites SBIR awards as an example of how the public and private sectors can collaborate in bringing new technology to markets. Further, see National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.

8

See response to NRC Phase II Survey, Question 23.

Not all valuable technology innovations lead directly to high revenues, however, and SBIR has led to valuable technology innovations across the spectrum of work funded by NIH. Technologies such as the bar-coding developed by Savi using SBIR funds have been of great importance to DoD, enabling massive savings in logistics. This degree of skew underscores the limitations of random surveys that can easily miss important outcomes in terms of mission and/or sales. SBIR companies that achieve significant sales and show promise of more are often acquired. Once acquired, they normally do not respond to surveys. For example, one firm, Digital Systems Resources, Inc., was acquired by General Dynamics in September of 2003. At the time of acquisition, DSR had received 40 Phase II SBIR awards and had reported $368 million in resultant sales and investment. See Findings and Recommendations chapter of National Research Council, An Assessment of the SBIR Program at the Department of Defense, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.

9

See response to NRC Phase II Survey, Question 12.

10

While 5.6 percent of respondents to the NRC Phase II Survey reported licensee sales greater than $0, three licenses reported more than $70 million in sales each and accounted for more than half of all reported sales, with one project alone reporting more than $200 million in licensee sales. See Section 4.2.3.4.

11

See Figure 4-5. The impact of the 2004 SBA ruling that excluded firms with majority ownership by venture firms is to be empirically assessed in a follow-on study by the National Research Council. On December 3, 2004, the Small Business Administration issued a final rule saying that to be eligible for an SBIR award “an entity must be a for-profit business at least 51 percent owned and controlled by one or more U.S. individuals.” U.S. Congress, House, Committee on Science, Subcommittee on Environment, Technology, and Standards, Hearing on “Small Business Innovation Research: What is the Optimal Role of Venture Capital,” Hearing Charter, June 28, 2005.

12

With SBIR funding, David Giuliani, together with University of Washington professors David Engel and Roy Martin, formed a company in 1987, eventually known as Optiva Corporation Inc., to promote a new, innovative dental hygiene device based on sonic technology. In October 2000, Royal Philips Electronics, acquired Optiva Corporation, Inc., now known as Philips Oral Healthcare. The final terms were not disclosed. Royal Philips Electronics, accessed at <http://www.homeandbody.philips.com/sonicare/gb_en/03d-story.asp>.

13

This assessment is based both on procedures for selecting topics and awardees aligned with agency mission, as well as outcomes analysis. Each of the five separate volumes prepared by the Committee on the SBIR program at the five individual agencies under review explores this issue in considerable depth. The Outcomes chapter of this volume summarizes these findings in more detail.

14

National Research Council, An Assessment of the SBIR Program at the Department of Defense, op. cit.

15

The growing interest of Defense prime contractors is recorded in National Research Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007.

16

National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

17

The importance of this strategy to NIH is reflected in the NIH RoadMap, announced in September 2003. Recent interest at NIH, led by the National Cancer Institute in conjunction with the SBIR program offices, has underscored the growth in interest in the role and potential of the program. See <http://nihroadmap.nih.gov/>.

18

The NIH Mission is “science in pursuit of fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability.”

19

The FDA describes its Critical Path Initiative as “an effort to stimulate and facilitate a national effort to modernize the scientific process through which a potential human drug, biological product, or medical device is transformed from a discovery or “proof of concept” into a medical product.” Accessed at <http://www.fda.gov/oc/initiatives/criticalpath/>.

20

National Research Council, An Assessment of the SBIR Program at the National Aeronautics and Space Administration, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.

21

NASA SBIR managers report that several SBIR projects relate to the Mars Rover. While these are considered to be valuable contributions to that mission, by definition they do not represent opportunities for high-volume sales. Interview with NASA Management, December 21, 2006.

22

National Research Council, An Assessment of the SBIR Program at the Department of Energy, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2008.

23

National Research Council, An Assessment of the SBIR Program at the National Science Foundation, op. cit.

24

NSF’s mission is “To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense.” Accessed at <http://www.nsf.gov/nsf/nsfpubs/straplan/mission.htm>.

25

Nearly 40 percent of U.S. scientific and technical workers are employed by small businesses. U.S. Small Business Administration, Office of Advocacy (2005) data drawn from U.S. Bureau of the Census; Advocacy-funded research by Joel Popkin and Company (Research Summary #211); Federal Procurement Data System; Advocacy-funded research by CHI Research, Inc. (Research Summary #225); Bureau of Labor Statistics, Current Population Survey; U.S. Department of Commerce, International Trade Administration.

26

For the breakout, see Table 4-7. Chapter 4 also includes a case study of Sociometrics, Inc., which illustrate the role of SBIR in firm formation.

27

See Figure 4-19 for a breakdown of the survey findings. Thirty-eight percent reported that they definitely would not have proceeded with the project without SBIR funding and an additional 33 percent reported that they probably would not have proceeded with the project without SBIR funding.

28

A significant number of firms involved in the program report that the founder was previously an academic. See Finding F-3.

29

See U.S. Small Business Administration, Tech-Net Database, accessed at <http://www.sba.gov/sbir/indextechnet.html>.

30

See Section 4.5.3 for a discussion of the incidence of new entrants. See also Figures 4-21, 4-22, 4-23, and 4-24 for data on new entrants at NSF, NIH, and DoD.

31

Academics represent an important future pool of applicants, firm founders, principal investigators, and consultants. Recent research shows that owing to the low number of women in senior research positions in many leading academic science departments, few women have the chance to lead a spinout. “Under-representation of female academic staff in science research is the dominant (but not the only) factor to explain low entrepreneurial rates amongst female scientists.” See Peter Rosa and Alison Dawson, “Gender and the commercialization of university science: academic founders of spinout companies,” Entrepreneurship & Regional Development, 18(4):341-366, July 2006.

32

See Chapters 3, 4, and 5 for additional discussion.

33

See Figure 2-4 and Figure 3-12 in National Research Council, An Assessment of the SBIR Program at the Department of Defense, op. cit.

34

See Figure 2-1 in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

35

See Section 3.2.6.4 in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

36

This statistic is drawn from the DoD Awards Database. See Figure 4-13.

37

See Figure 4.2-15 and Figure 4.2-16 in National Research Council, An Assessment of the SBIR Program at the National Science Foundation, op. cit.

38

See Figure 4.2-14 in National Research Council, An Assessment of the SBIR Program at the National Science Foundation, op. cit.

39

See Figures 2-1, 2-2, 3-10, and 3-11 in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit.

40

See Table 4-23 in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit. The patenting activity of SBIR firms has also been highlighted in recent Congressional testimony. Representatives of the Small Business Technology Council claimed that SBIR-related companies produced some 45,000 patents, compared to just over 2,900 patents produced by universities in 2006. The same testimony, citing the National Science Foundation, reported that some 32 percent of U.S. scientists and engineers are employed by small businesses. (Testimony of Michael Squillante before the Senate Committee on Small Business and Entrepreneurship on July 12, 2006) The Small Business Administration reports that small businesses employ 41 percent of high-tech workers, including scientists, engineers, and computer workers. U.S. Small Business Administration, Frequently Asked Questions, June 2006, accessed at <http://www.sba.gov/advo/stats/sbfaq.pdf>.

41

See Section 4.6.3.1 on university faculty and company formation. Also see Table 4-13 on university involvement in SBIR projects.

42

See also National Research Council, An Assessment of the SBIR Program at the National Science Foundation, op. cit.

43

For a discussion of indirect path effects, see National Research Council, The Advanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2001.

44

See the discussion of agencies differing goals, modes of operation, and their implications for the assessment in the Introduction chapter. See also the section on assessment challenges, and for a more complete review, see the study’s methodology report, National Research Council, An Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004.

45

The difference between grants and contracts is discussed in Finding H.

46

For a discussion of expanded award size and flexibility at NIH, see the Awards chapter in National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit. For a discussion of the Phase IIB awards at NSF, see the Program Management chapter in National Research Council, An Assessment of the SBIR Program at the National Science Foundation, op. cit.

47

See the discussion of the challenges of early-stage finance in the Introduction to this volume.

48

The Small Business Technology Transfer (STTR) was established by Congress in 1992 with a similar statutory purpose as SBIR. A major difference between SBIR and STTR is that the STTR requires the small business to have a research partner consisting of a university, a Federally Funded Research and Development Center (FFRDC), or a qualified nonprofit research institution. In STTR, the small business must be the prime contractor and perform at least 40 percent of the work, with the research partner performing at least 30 percent of the work. The balance can be done by either party and/or a third party. Currently, the STTR set-aside is 0.3 percent of an agency’s extramural R&D budgets.

49

See the presentation, “The Private Equity Continuum,” by Steve Weiss, Executive Committee Chair of Coachella Valley Angel Network, at the Executive Seminar on Angel Funding, University of California at Riverside, December 8-9, 2006, Palm Springs, CA. In a personal communication, Weiss points out the critical contributions of SBIR to the development of companies such as Cardio-Pulmonics. The initial Phase I and II SBIR grants allowed the company to demonstrate the potential of its products in animal models of an intravascular oxygenator to treat acute lung infections and thus attract angel investment and subsequently venture funding. Weiss cites this case as an example of how the public and private sectors can collaborate in bringing new technology to markets. Steve Weiss, Personal Communication, December 12, 2006.

50

There may be substantially more venture funding than the NRC research has, as yet, revealed. The GAO report on venture funding within the NIH and DoD SBIR programs used a somewhat different methodology to identify firms with VC funding. As a result of the approach adopted, no conclusions can be drawn from the study as to whether firms identified as VC funded are in fact excluded from the SBIR program on ownership grounds. In addition, the number of VC-funded firms—reportedly 18 percent of all NIH firms receiving Phase II awards from 2001-2004—is considerably higher than suggested by preliminary NRC analysis. U.S. Government Accountability Office, Small Business Innovation Research: Information on Awards made by NIH and DoD in Fiscal years 2001-2004, GAO-06-565, Washington, DC: U.S. Government Accountability Office, 2006.

51

Access the SBA’s 2002 SBIR Policy Directive, Section 3(y)(3) at <http://www.zyn.com/sbir/sbres/sba-pd/pd02-S3.htm>.

52

This new interpretation of “individuals” resulted in the denial by the SBA Office of Hearings and Appeals of an SBIR grant in 2003 to Cognetix, a Utah biotech company, because the company was backed by private investment firms in excess of 50 percent in the aggregate. Access this decision at <http://www.sba.gov/aboutsba/sbaprograms/oha/allcases/sizecases/siz4560.txt>. The ruling by the Administrative Law Judge stated that VC firms were not “individuals,” i.e., “natural persons,” and therefore SBIR agencies could not give SBIR grants to companies in which VC firms had a controlling interest. The biotechnology and VC industries have been dismayed by this ruling, seeing it as a new interpretation of the VC-small business relationship by SBA. See for example, testimony by Thomas Bigger of Paratek Pharmaceuticals before the U.S. Senate Committee on Small Business and Entrepreneurship, July 12, 2006.

53

This paragraph is a correction of the text in the prepublication version released on July 27, 2007.

54

The 1994-2004 period saw a decline in venture investments in seed and early stage and a concomitant shift away from higher-risk early-stage funding. The last few years have seen some recovery in early-stage finance. See the discussion of this point in the Introduction chapter. See also National Science Board, Science and Technology Indicators 2006, Arlington, VA: National Science Foundation, 2006. This decline is reportedly particularly acute in early-stage technology phases of biotechnology where the investment community has moved toward later-stage projects, with the consequence that early-stage projects have greater difficulty raising funds. See the testimony by Jonathan Cohen, founder and CEO of 20/20 GeneSystems, at the House Science Committee Hearing on “Small Business Innovation Research: What is the Optimal Role of Venture Capital,” July 28, 2005.

55

See the statements by Ron Cohen, CEO of Acorda Technologies, and Carol Nacy, CEO of Sequella Inc, at the House Science Committee Hearing on “Small Business Innovation Research: What is the Optimal Role of Venture Capital,” July 28, 2005. Squella’s Dr. Nacy’s testimony captures the multiple sources of finance for the 17-person company (June 2005). They included—founder equity investments; angel investments; and multiple, competitive scientific research grants, including SBIR funding for diagnostics devices, vaccines, and drugs. SBIR funding was some $6.5 million out of a total of $18 million in company funding. Dr. Nacy argues that SBIR funding focuses on research to identify new products while venture funding is employed for product development.

56

The biomedical context of this finding on venture capital funding reflects the recent attention to this issue in the NIH SBIR program.

57

See the testimony by Jonathan Cohen, founder and CEO of 20/20 GeneSystems, at the House Science Committee Hearing on “Small Business Innovation Research: What is the Optimal Role of Venture Capital,” July 28, 2005. In the same hearing Mr. Fredric Abramson, President and CEO of AlphaGenics, Inc., argues that “any change that permits venture owned small business to compete for SBIR will jeopardize biotechnology innovation as we know it today.” For additional arguments against including VC-backed firms in the SBIR program, see testimony by Robert Schmidt at the House Science Committee Subcommittee on Technology and Innovation Hearing on “Small Business Innovation Research Authorization on the 25th Program Anniversary,” April 26, 2007.

58

Included in this group of firms are highly successful SBIR companies like Invitrogen, MedIm-mune, and Martek. For discussion of the factors affecting the returns to venture capital organizations, including incentive and information problems and the role venture funds have played in supporting a limited number of highly successful firms, see Paul Gompers and Josh Lerner, The Venture Capital Cycle, Cambridge, MA: The MIT Press, 2000, Ch. 1.

59

See National Venture Capital Association, Money Tree Report, November, 2006. The mean venture capital deal size for the first three quarters of 2006 was $8.03 million. This trend has been accelerated by the growth of larger venture firms. See Paul Gompers and Josh Lerner, The Venture Capital Cycle, op. cit.

60

Firms that have used SBIR in this manner include Neurocrine and Illumina. The latter indicated in interviews that these alternative paths later become critical products that underpinned the success of the company. See also Dr. Nacy’s testimony cited above.

61

This research will address questions such as—which NIH SBIR participating companies have been or are likely to be excluded from the program as a result of the 2002 rule change on Venture Capital Company ownership; and what is the likely impact of the 2002 ruling had it been applied during the 1992-2006 timeframe and what is its probable current impact? Key variables will include the presence and amount of SBIR support, the receipt of venture capital funding or other outside funding, and output measures including those related to commercialization and knowledge generation.

62

DoD has been particularly diligent in seeking outside evaluations of its program’s operations and innovations, e.g., the Fast Track initiative, evaluated in 2000 and again in 2006 in a further study by the NSF. Similarly, NIH recently commissioned outside evaluation by the NRC to assess the impact of the SBA ruling on venture-funded firms.

63

The recent increase in the average and median size of Phase I and Phase II awards provides an example. NIH staff has offered a number of different justifications, but no systematic analysis or review appears to have preceded such an important change to the program.

64

According to recent OECD analysis, the International Benchmark for program evaluation of large SME and Entrepreneurship Programs is between 3 percent (for small programs) and 1 percent (for large-scale programs). See Organization for Economic Cooperation and Development, “Evaluation of SME Policies and Programs: Draft OECD Handbook,” OECD Handbook CFE/SME(2006)17. See the recommendation for supplementary management funding.

65

See U.S. General Accounting Office, Federal Research: Small Business Innovation Research Participants Give Program High Marks, Washington, DC: U.S. General Accounting Office, 1987; U.S. General Accounting Office, Federal Research: Assessment of Small Business Innovation Research Program, Washington, DC: U.S. General Accounting Office, 1989; U.S. General Accounting Office, Small Business Innovation Research Program Shows Success But Can Be Strengthened, RCED–92–32, Washington, DC: U.S. General Accounting Office, 1992; U.S. General Accounting Office, Federal Research: DoD’s Small Business Innovation Research Program, RCED–97–122, Washington, DC: U.S. General Accounting Office, 1997; U.S. General Accounting Office, Federal Research: Evaluations of Small Business Innovation Research Can Be Strengthened, RCED–99–198, Washington, DC: U.S. General Accounting Office, 1999. U.S. General Accountability Office, Small Business Innovation Research: Information on Awards made by NIH and DoD in Fiscal Years 2001 through 2004, GAO-06-565, Washington, DC: U.S. General Accountability Office, 2006. U.S. General Accountability Office, Small Business Innovation Research: Agencies Need to Strengthen Efforts to Improve the Completeness, Consistency, and Accuracy of Awards Data, GAO-07-38, Washington, DC: U.S. General Accountability Office, 2006. See also Bruce Held, Thomas Edison, Shari Lawrence Pfleeger, Philip Anton, and John Clancy, Evaluation and Recommendations for Improvement of the Department of Defense Small Business Innovation Research (SBIR) Program, Arlington, VA: RAND National Defense Research Institute, 2006.

66

The agencies should consider providing data from a range of sources—including agency databases, agency surveys, the patent office, and bibliographic databases, along with data from award recipients themselves.

67

As noted above, a recent OECD report, the International Benchmark for program evaluation of large SME and Entrepreneurship Programs is between 3 percent for small programs and 1 percent for large-scale programs. See “Evaluation of SME Policies and Programs: Draft OECD Handbook,” op. cit.

68

Each of these options has its advantages and disadvantages. For the most part, over the past 25 years, the departments, institutes, and agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legislation does not permit this and would have to be modified. This would also limit funds for awards to small companies, the program’s core objective. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an overall increase in the program. In any case, the Committee envisages an increase of the “set-aside” of perhaps 0.03 percent to 0.05 percent on the order of $35 million-40 million per year, or roughly double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set aside” to 2.55 percent, providing modest resources to assess and manage a program that is approach ing an annual spend of some $2 billion. Whatever modality adopted by Congress, the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot without the benefit of additional resources.

69

Drawn from case study interviews and gap data from the NRC Phase II Survey.

70

See case studies at all agencies. In addition, the NRC Phase II Survey indicated that most companies had stopped work in between phases, and many had experienced other funding gaps. See Chapter 5: Program Management.

71

This recommendation should not be interpreted as lowering the bar for the acceptance of proposals from woman- and minority-owned companies, but rather as assisting them to become able to meet published criteria for grants at rates similar to other companies on the basis of merit, and to ensure that there are no negative evaluation factors in the review process that are biased against these groups.

72

The intent here is to use the DSB or the NSF Board as a model, not something necessarily to be copied exactly.

73

Discussions with NIH SBIR program managers, June 13, 2006.

74

Phase I awards may have particular importance in meeting noncommercial objectives of the program, for example, helping academics to transition technologies out of the lab into startup companies.

75

National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., pp. 33-39.

76

For a discussion of Phase III commercialization issues, see National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.

77

Randomized evaluations are recognized as a highly rigorous means for evaluating the effectiveness of a strategy or approach across many diverse fields. When properly conceived and executed, they enable one to determine to a high degree of confidence whether the new approach itself, as opposed to other factors, causes the observed outcomes. Such randomized evaluations are recognized as an effective means for evaluating an intervention’s effectiveness across many diverse fields. When properly conceived and executed, they enable one to determine with some confidence whether the intervention itself, as opposed to other factors, causes the observed outcomes.

See, for example, U.S. Department of Education, “Scientifically-Based Evaluation Methods: Notice of Final Priority,” Federal Register, 70(15):3586-3589, January 25, 2005; the Food and Drug Administration’s standard for assessing the effectiveness of pharmaceutical drugs and medical devices, at 21 C.F.R. §314.12; “The Urgent Need to Improve Health Care Quality,” Consensus statement of the Institute of Medicine National Roundtable on Health Care Quality, Journal of the American Medical Association, 280(11):1003, September 16, 1998; American Psychological Association, “Criteria for Evaluating Treatment Guidelines,” American Psychologist, 57(12):1052-1059, December 2002; Society for Prevention Research, Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination, April 12, 2004, at <http://www.preventionresearch.org/softext.php>; Office of Management and Budget, What Constitutes Strong Evidence of Program Effectiveness, pp. 4-8, accessed at <http://www. whitehouse.gov/omb/part/2004_program_eval.pdf, 2004 >.

78

Accessed at <http://officeofbudget.od.nih.gov/PDF/BRDPI_2_5_07.xls>.

79

For example, a 25 percent increase in the size of Phase II awards implies a 25 percent reduction in the number of Phase II awards, all other things being equal—or a much larger reduction in the number of Phase I awards.

80

Recognizing that these values are not identical, the erosion by inflation of the amounts available for Phase I appear more constraining than for Phase II. One may argue that, for parity, the Phase II should be raised to approximately $1,125,000; the trade-offs involve attracting adequate numbers of quality proposals, and providing sufficient resources to enable firms to actually carry out the necessary work, while also recognizing the impact on the number of awards. While recognizing that there is necessarily an arbitrary element in fixing these amounts, the Committee is confident that a $1,000,000 Phase II award will maintain the program’s attraction to innovative small businesses.

81

Program innovations that offer flexibility in award size introduced at NIH appear promising, though there are currently no data to determine whether this approach has resulted in more successful projects. Several agencies, NIH, NSF, and NASA, have successfully made their own judgments as to the most appropriate award sizes to meet their agency needs and continue to draw adequate “deal flows.” Agencies with smaller SBIR programs, not reviewed here, have adopted much smaller award sizes.

82

See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, op. cit., Awards chapter. See also the discussion of the Phase IIB program in National Research Council, An Assessment of the SBIR Program at the National Science Foundation, op. cit.

83

See Section 3.8 and Section 4.4.4.2 for related data and discussion.

84

Creare, Inc., has received a significant number of SBIR awards and now has 21 patents resulting from SBIR-funded work. Staff members have published dozens of papers. The firm has licensed technologies including high-torque threaded fasteners, a breast cancer surgery aid, corrosion preventative coverings, an electronic regulator for firefighters, and mass vaccination devices (pending). Products and services developed at Creare include thermal-fluid modeling and testing, miniature vacuum pumps, fluid dynamics simulation software, network software for data exchange. A significant technical accomplishment was achieved with the NCS Cryocooler used on the Hubble Space Telescope to restore the operation of the telescope’s near-infrared imaging device.

In some cases, the company has developed technical capabilities that have remained latent for years until a problem arose for which those capabilities were required. The cryogenic cooler for the Hubble telescope is an example. The technologies that were required to build that cryogenic refrigerator started being developed in the early 1980s as one of Creare’s first SBIR projects. Over 20 years, Creare received over a dozen SBIR projects to develop the technologies that ultimately were used in the cryogenic cooler.