4.1.1. Compared to What?

Assessment usually involves comparison—comparing programs and activities, in this case. Three kinds of comparison seem possible: with other programs at each agency, between SBIR programs at the various agencies, and with early-stage technology development funding in the private sector, such as venture capital activities. Yet, the utility of each of these three types of comparison is limited.

Other award programs at the agencies have fundamentally different objectives, such as promoting basic research through grant programs (at the National Institutes of Health [NIH] and the National Science Foundation [NSF]), developing capacity (awards for university infrastructure), or training. There are often no other dedicated programs for innovative small businesses. And no other programs for small businesses have as a primary goal the commercial exploitation of research. This fundamental difference in objectives makes it difficult to usefully compare an SBIR program with other programs at the relevant agency.

Comparisons between SBIR programs at different agencies appear superficially more useful, but must be regarded with considerable caution. As discussions in Chapter 1 of this volume and in the separate agency volumes indicate, the widely differing agency missions have shaped the agency SBIR programs, focusing them on different objectives and on different mechanisms and approaches. Agencies whose mission is to develop technologies for internal agency use via procurement—notably the Department of Defense (DoD) and the National Aeronautics and Space Administration (NASA)—have a quite different orientation from agencies that do not procure technology and are instead focused on developing technologies for use outside the agency.

Finally, SBIR might be compared with venture capital (VC) activities, but there are important differences. VC funding is typically supplied later in the development cycle when innovations are in, or close to, market—most venture investments are made with the expectation of an exit from the company within three years. VC investments are typically larger than SBIR awards—the average investment made by VC firms in a company was over $7 million in 2005, compared to less than $1 million for SBIR over a two to three year cycle.³ VC investments are also focused on companies, not projects, and often come both with substantial management support and influence (e.g., through seats on the company’s board). None of this is true for SBIR.

BOX 4-1Multiple Sources of Bias in Survey Response

Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below.^a

• Successful and more recently funded firms are more likely to respond. Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and it increased the greater the award amount.^b Nearly 40 percent of respondents in the NRC Phase II Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid-1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards.
• Success is self reported. Self-reporting can be a source of bias, although the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospectively analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third party observation.^c In short, company founders and/or principal investigators are in many cases simply the best source of information available.
• Survey sampled projects at firms with multiple awards. Projects from firms with multiple awards were underrepresented in the sample, because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards.
• Failed firms are difficult to contact. Survey experts point to an “asymmetry” in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.^d It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists.
• Not all successful projects are captured. For similar reasons, the NRC Phase II Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment.
• Some firms may not want to fully acknowledge SBIR contribution to project success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a variety of reasons. For example, some may understandably attribute success exclusively to their own efforts.
• Commercialization lag. While the NRC Phase II Survey broke new ground in data collection, the amount of sales made—and indeed the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from NIH SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50 percent greater than can be captured in a single survey.^e This underscores the importance of follow-on research based on the now-established survey methodology.

FIGURE B-4-1

Survey bias due to commercialization lag.

These sources of bias provide a context for understanding the response rates to the NRC Phase I and Phase II Surveys conducted for this study. For the NRC Phase II Survey, of the 4,523 firms that could be contacted out of a sample size of 6,408, 1,916 responded, representing a 42 percent response rate. The NRC Phase I Survey captured 10 percent of the 27,978 awards made by all five agencies over the period of 1992 to 2001. See Appendix A and B for additional information on the surveys.

a

For a technical explanation of the sample approaches and issues related to the NRC surveys, see Appendix A.

b

Albert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005.

c

While economic theory is formulated on what is called “revealed preferences,” meaning individuals and firms reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See {1} Gregory G. Dess and Donald W. Beard, “Dimensions of Organizational Task Environments,” Administrative Science Quarterly, 29: 52-73, 1984; {2} Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998.

d

Albert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, op. cit.

e

Data from NIH indicates that a subsequent survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market, and in the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charels W Wessner, ed., Washington, DC: The National Academies Press, 2009.

The lack of available comparators means that we must assess each program in terms of the benchmarks developed to review the program in the Methodology report described below.⁴

4.2.1. Challenges of Commercialization

Commercialization of the technologies developed under the research supported by SBIR awards has been a central objective of the SBIR program since its inception. The program’s initiation in the early 1980s, in part, reflected a concern that American investment in research was not adequately deployed to the nation’s competitive advantage. Directing a portion of federal investment in R&D to small businesses was thus seen as a new means of meeting the mission needs of federal agencies, while increasing the participation of small business and thereby the proportion of innovation that would be commercially relevant.⁵

Congressional and Executive branch interest in the commercialization of SBIR research has increased over the life of the program.

A 1992 GAO study⁶ focused on commercialization in the wake of congressional expansion of the SBIR program in 1986.⁷ The 1992 reauthorization specifically “emphasize[d] the program’s goal of increasing private sector com mercialization of technology developed through federal research and development.”⁸ The 1992 reauthorization also changed the order in which the program’s objectives are described, moving commercialization to the top of the list.⁹

The term “commercialization” means “reaching the market,” which some agency managers interpret as “first sale”—that is, the first sale of a product in the market place, whether to public or private sector clients. This definition, however, misses significant components of commercialization that do not result in a discrete sale. It also fails to provide any guidance on how to evaluate the scale of commercialization, an important element in assessing the degree to which SBIR programs successfully encourage commercialization. The metrics for assessing commercialization can also be elusive,¹⁰ and it is important to understand that it is not possible to completely quantify all commercialization from a research project:

• The multiple steps needed after the research has been concluded mean that a single, direct line between research inputs and commercial outputs rarely exists in practice; cutting edge research is only one contribution among many leading to a successful commercial product.
• Markets themselves have major imperfections, or information asymmetries so high quality, even path-breaking research, does not always result in commensurate commercial returns.
• The lags involved in the timeline between an early stage research project and a commercial outcome mean that for a significant number of the more recent SBIR projects, commercialization is still in process, and sales—often substantial sales—will be made in the future. The current “total” sales are in this case just a “snapshot half way through the race,” and will require updating as the full impact of the award becomes apparent in sales.
• Yet the impact of SBIR awards also needs to be qualified. Research rarely results in stand-alone products. Often, the output from an SBIR project is combined with other technologies. The SBIR technology may provide a critical element in developing a winning solution, but that commercial impact—the sale of the larger combined product—is not captured in the data.
• In some cases, the full value of an “enabling technology” that can be used across industries is difficult to capture.

All this is to say that commercialization results must be viewed with caution, first because our ability to track them is limited (indeed it appears highly likely that our efforts at quantification of research awards may understate the true commercial impact of SBIR projects) and because an award, and a successful project, cannot lay claim to all subsequent commercial successes, though it may contribute to that success in a significant fashion.

These caveats notwithstanding, it is possible to deploy a variety of assessment techniques to measure commercialization outcomes.

4.2.2. Commercialization Indicators and Benchmarks

This report uses three sets of indicators to quantitatively assess commercialization success:

1. Sales and Licensing Revenues (“sales” hereafter, unless otherwise noted). Revenues flowing to a company from the commercial marketplace and/or through government procurement constitute the most obvious measure of commercial success. They are also an important indicator of uptake for the product or service. Sales indicate that the result of a project has been sufficiently positive to convince buyers that the product or service is the best available solution.
   Yet if there is general agreement that sales are a key benchmark, there is no such agreement on what constitutes “success.” Companies, naturally enough, focus on projects that contribute to the bottom line—that are profitable. Agency staff provide a much wider range of views. Some view any sales a substantial success for a program focused on such an early stage of the product and development cycle, while others seem more ambitious.¹¹ Some senior executives in the private sector viewed only projects that generated cumulative revenues at $100 million or more as a complete commercial success.¹²
   Rather than seeking to identify a single sales benchmark for “success,” it therefore seems more sensible to simply assess outcomes against a range of benchmarks reflecting these diverse views, with each marking the transition to a greater level of commercial success:
   1. Reaching the market—a finished product or service has made it to the marketplace
   2. Reaching $1 million in cumulative sales (beyond SBIR Phases I and II)—the approximate combined amount of standard DoD Phase I and Phase II awards
   3. Reaching $5 million in cumulative sales—a modest commercial success that may imply that a company has broken even on a project
   4. Reaching $50 million in cumulative sales—a full commercial success.
2. Phase III Activities Within DoD. As noted above, Phase III activities within DoD are a primary form of commercialization for DoD SBIR projects. These activities are considered in Section 4.3 (Agency Mission) of this report and Chapter 5 (Phase III Challenges and Opportunities) of the DoD report.¹³
3. R&D Investments and Research Contracts. Further R&D investments and contracts are good evidence that the project has been successful in some significant sense. These investments and contracts may include partnerships, further grants and awards, or government contracts. The benchmarks for success at each of these levels should be the same as those above, namely:
   1. Any R&D additional funding
   2. Additional funding of $1 million or more
   3. Additional funding of $5 million or more
   4. Funding of $50 million or more
4. Sale of Equity. This is a clear-cut indicator of commercial success or market expectations of value. Key metrics include:
   1. Equity investment in the company by an independent third party
   2. Sale or merger of the entire company

4.2.3. Sales and Licensing Revenues

The most basic of all questions on commercialization is whether results from a project reached the marketplace. The NRC Phase II Survey¹⁴ indicates that just under half (47 percent) of respondents had generated some sales, and that a further 18 percent still expected sales, though they had none at the time of the survey. In addition, 5 percent were still in the research stage of the project. There is variation among the agencies, but these data are consistent with program objectives.¹⁵

FIGURE 4-1

Sales from Phase II projects.

SOURCE: NRC Phase II Survey.

4.2.3.1. Distribution of Sales

Research on early-stage financing strongly suggests a pronounced skew to the results, and this turns out to be the case. Most projects that reach the market generate minimal revenues. A few awards generate substantial results, and a small number bring in large revenues.

Of the 790 SBIR Phase II projects reporting sales greater than $0, average sales per project were $2,403,255. Over half of the total sales dollars were due to 26 projects (1.4 percent of the total), each of which had $15,000,000 or more in sales. The highest cumulative sales figure reported was $129,000,000.

This distribution is reflected in Figure 4-2.

FIGURE 4-2

Distribution of projects with sales >$0. SOURCE: NRC Phase II Survey.

Almost three quarters of the projects reporting sales greater than zero had $1 million or less in sales; two projects reported sales greater than $100 million. The latter by themselves accounted for 16.5 percent of all the revenues reported; together, the 1.7 percent of respondents reporting sales greater than $20 million accounted for 43.7 percent of all revenues reported.

These distributions are similar to those reported from other SBIR data sources.¹⁶ For example, at NIH, sales data are available from the NRC survey, a previous NIH survey, and the DoD commercialization database.¹⁷ These data in Figure 4-4 show that about 10 percent of NIH projects that report any revenues report more than $5 million.

FIGURE 4-4

NIH sales by sales range (percent of projects in each range). Total for 1992-2002. SOURCE: NIH data, NRC Phase II Survey, and DoD Commercialization database.

FIGURE 4-3

Distribution of sales, by total sales (percent of total sales dollars).

SOURCE: NRC Phase II Survey.

4.2.3.2. Sales Expectations

Because it may take years after the end of a Phase II for a commercial product to reach the market, many projects that do not yet report sales still expect them eventually. About 36 percent of NRC survey respondents with no sales (19 percent of all projects) still expected sales in the future. Analysis elsewhere suggests that these expectations are often optimistic, and that a considerably smaller number of these projects will, in the end, reach the market.

However, it is equally important to note that a complete accounting of all sales from the projects funded during 1992-2001 (the focus of the NRC survey) will be possible only some years in the future. Many projects have only recently reached the market, so the bulk of their sales will be made in the future and are not captured in these survey data, which effectively capture initial sales (see Box 4-2).

BOX 4-2

Underestimating Commercial Outcomes from the SBIR Program: The Impact of Systematic Characteristics of Survey Analysis. Among the SBIR agencies, only DoD requires that firms enter commercialization data into a database when applying for subsequent awards. (more...)

TABLE 4-1Most Important Customer (Percent of Responses)

View in own window

Most Important Customer	Percent of Responses
Domestic private sector	35
Department of Defense (DoD)	32
Prime contractors for DoD or NASA	10
NASA	2
Other federal agencies	1
State or local governments	4
Export markets	14
Other	2

SOURCE: NRC Phase II Survey.

4.2.3.4. Licensee Sales and Related Revenues

The indirect impact of licensee sales, where survey respondents report sales not made by their own company, is an important measure of success. These data are important as an indicator of the extended effects of SBIR beyond the immediate awardee company. However, they should be treated with an additional degree of caution, as respondents do not necessarily have as accurate information about another company as they have about their own.

Licensing activity within the program is significant: Just over 35 percent reported a finalized licensing agreement or ongoing negotiations towards one.¹⁹

Just over 5 percent of respondents reported licensee sales greater than $0, with three licenses reporting more than $70 million in sales each and accounting for more than half of all reported licensee sales. One project alone reported more than $200 million in licensee sales.

As the case study in Box 4-3 indicates, in some cases licensing has been pivotal in the commercialization of a successful technology: In some industries, there are no alternatives to using established channels.

BOX 4-3

Licensing Case Study. Applied Health Science and the Wound and Skin Intelligence System™ Applied Health Science (AHS) received a Phase I award to validate and automate the Pressure Sore Status Tool, a standardized assessment instrument for use (more...)

4.2.4. Additional Investment Funding

Further investment in a recipient company related to the SBIR award project is another indication that the project work is of value. On average, SBIR projects received almost $800,000 from non-SBIR sources, with over half of respondents (51.6 percent) reporting some additional funds for the project from a non-SBIR source.²⁰

Focusing just on the 839 projects that reported receiving more than $0 in additional funding, these projects reported average additional funding of $1,538,438, with almost $260,000 from federal agencies themselves, mostly from DoD. Three hundred eighteen projects reported some federal funding, averag ing just under $1.6 million per project. At DoD, just over 30 percent of projects (205) reported additional federal funding outside SBIR, averaging just over $1.6 million per project.

About $165,000 per project came from venture capital sources. However, in those cases where venture funding was present, the amounts of funding were substantial: For the 50 responding projects with VC funding, the average per project was just under $8.3 million.

These figures reflect the well-known concentration of venture funding on a few, highly desirable projects. They also show that SBIR supports a wide range of projects which do have commercial prospects (as well as other possible benefits) that are not likely to be funded by venture capital.

Focusing more closely on venture funding at NIH, initial research indicates that approximately 50 of the 200 companies that won the most Phase II awards at NIH have received some venture funding.²¹ This is reflected in Figure 4-5.

FIGURE 4-5

Distribution of venture funding for NIH top 200 Phase II winners. SOURCE: VentureSource and other venture capital databases; National Institutes of Health awards database. See also National Research Council, An Assessment of the SBIR Program at the National (more...)

Total VC investment is approximately $1.59 billion in these 50 companies, a total that dwarfs the $272 million NIH SBIR investment in these companies.

In addition, the NRC Firm Survey determined that 15 firms had had initial public offerings, and that a further six firms planned such offerings for 2005-2006. SBIR firms also generate a significant number of new companies. Fourteen percent of responding firms indicated they had formed a spin-off company, with a total of 242 spin-off companies reported.

In contrast to the tightly concentrated distribution of venture funding, inter nal funding was by far the most widespread form of support, being reported by almost 50 percent of all respondents. Average internal funding ($113,000), was much lower than average venture funding.

About 7 percent of all respondents reported receiving “other private equity.” This funding averaged $1.9 million per project for those projects receiving this kind of funding. Investments from government and academic sources were relatively scarce (less than 5 percent of respondents), and provided relatively small amounts (just over $225,000 per respondent receiving these funds).

4.2.5. Additional SBIR Funding

Aside from providing non-SBIR funds, the federal government in many cases makes further investments via the SBIR program itself. The NRC surveys asked respondents how many additional Phase I and Phase II awards followed each initial award, related to the original project.

About 40 percent of respondents reported receiving at least one additional related Phase II award, and slightly over half (53.8 percent) reported at least one additional Phase I award.

Relatively few projects received many related awards: Only 8.4 percent of respondents reported at least 5 related Phase I awards, and 9.7 percent received at least three related Phase IIs. This suggests that the “clustering” hypothesis—that SBIR type projects often require multiple awards, sometimes looping back to Phase I, before reaching the market—only applies to a limited number of projects, although case studies (e.g., SAM Technologies²²) indicate that there are also important cases where such clustering does occur. In short, the data suggest that some companies and projects do attract a cluster of SBIR awards, but most do not. It may be, however, that this concentration of clustered awards reflects the limited number of commercial successes as well. Just under one half of respondents (47 percent) reported no additional related Phase I awards, and 60 percent reported no related Phase II.

4.2.6. SBIR Impact on Further Investment

Both the NRC and NIH Surveys sought additional information about the impact of the SBIR program on company efforts to attract third party funding—the “halo effect” mentioned by some interviewees, who suggested that an SBIR award acted as a form of validation for external inventors.²³

The fact that two-thirds of SBIR respondents did not attract outside funding, and that only 3.5 percent received venture funding, suggests that receiving a Phase II SBIR award does not in itself guarantee external funding. Survey responses from other surveys did, however, paint a more positive picture of these effects: 78 percent of NIH respondents said that they believed that additional capital had “resulted from” their SBIR participation—a strong statement.²⁴

Case study interviews provided mixed views on this, with some interviewees strongly supporting the view that SBIR helps to attract investment and others suggesting that SBIR awards had had relatively little impact, although these views obviously reflect individual company experiences.

4.2.7. Small Company Participation and Employment Effects

Employment is another indicator of commercial success and also that the program is supporting small business.

The median size of a company receiving SBIR awards is relatively small—far lower than the 500-employee limit imposed by the SBA (see Figure 4-6).

FIGURE 4-6

Distribution of companies, by number of employees at time of award. SOURCE: NRC Phase II Survey.

The program focuses the bulk of its awards on very small companies. More than a third of awardees had between one and five employees at the time of award. A very substantial number (seventy percent) of respondent companies had 20 employees or fewer at the time of the Phase II award.

The NRC Survey sought detailed information about the number of employees at the time of the award and at the time of the survey and about the direct impact of the award on employment. Overall, the survey data showed that the average employment gain at each responding firm from the date of the SBIR award to the time of the survey was 29.9 full-time equivalent employees. Of course, very few of the companies that went out of business responded to the survey, so this question is particularly skewed toward firms that have been at least somewhat successful.

Most responding companies have expanded since the date of the Phase II award. The NRC Phase II Survey also shows that respondents enjoyed strongly positive employment growth after receiving a Phase II award. Table 4-4 shows that the percentage of companies with at least 50 employees more than doubled, from 16.5 percent to 35.4 percent of all respondents. Overall, survey respondents reported gains of 57,808 full time equivalent employees, with the top five respondents accounting for 18.4 percent of the overall net gain.

TABLE 4-4

Employment at Phase II Respondent Companies, at the Time of Award and Currently.

The NRC survey also sought to directly identify employment gains that were the direct result of the award. Respondents estimated that specifically as a result of the SBIR project, their firm was able to hire an average of 2.4 employees, and to retain 2.1 more.²⁵

FIGURE 4-7

Employment distribution at responding companies, at time of award and currently.

SOURCE: NRC Phase II Survey.

4.2.8. Sales of Equity and Other Company-level Activities

Company-level activities offer another set of indicators for measuring commercial activity, as these suggest that something of commercial value is being developed, even if no sales have as yet resulted. For example, although Neurocrine, an NIH SBIR awardee has yet to produce a product, it achieved an IPO that raised more than $100 million.²⁶

The NRC Phase II Survey explored whether SBIR awardees had finalized agreements or ongoing negotiations on various company-level activities. The data show that marketing-related activities were most widespread. Activities with foreign partners were lower than similar activities with U.S. partners. Note, however, that the question asked specifically for outcomes that were the “result of the technology developed during this project”—a tight description.²⁷

The impact of these activities is hard to gauge using quantitative assessment tools only. Box 4-4 illustrates how research conducted using SBIR funding seeded an entire generation of spin-off companies and joint ventures in a technology of potential significance for homeland security.

BOX 4-4

Intelligent Optical Systems: Intelligent, Distributed, Sensitive Chemical and Biochemical Sensors and Sensor Networks. Intelligent Optical Systems has developed a system for using the entire length of a specially designed fiber-optic cable as a sensor (more...)

4.2.9. Commercialization: Conclusions

While accepting the view that there is no single, simple metric for determining the commercial success of an early stage R&D program such as SBIR, numerous metrics do provide the basis for making a broad determination of commercial outcomes at SBIR.

These data, taken together, strongly support the view that the program has a strong commercial focus, with considerable efforts to bring projects to market, with some success. Even though the number of major commercial successes has been few, that is normal for early stage high-risk projects, and the overall commercialization effort is substantial. Products are coming to market quickly, significant licensing and marketing efforts are under way for many projects, and approximately 30-40 percent of projects generate products that do reach the marketplace. These data all paint a picture of a program that is successful in commercializing innovative technologies in a variety of ways.

4.3.1. Procedural Alignment of SBIR Programs and Agency Mission

A procedural assessment reviews the steps taken by each agency program to ensure that the design and procedures of their SBIR program are aligned with the needs of the agency.

Agencies do this in different ways, but the following areas of analysis are broadly shared by all the agencies.

4.3.1.2. Topic Development Process—DoD

The agencies—notably DoD but, to a lesser extent, all the SBIR agencies—have an elaborate process for developing proposals for topics, sifting and assessing these proposals, and then finally deciding which topics should be published in the solicitation.

Agency procedures differ, and are described in the agency volumes. At DoD, for example, the topic selection process includes multiple levels of review. While this process has been criticized for adding considerable time between the initial identification of an agency’s needs and the first dollars flowing to those needs, the process also seeks to ensure that all of the stakeholders feel that they have had a say. This is particularly important for DoD because it has a number of different stakeholder communities.

FIGURE 4-8

Topic review process.

SOURCE: NRC chart developed from interviews with Department of Defense staff.

Ultimate decision authority on the inclusion of topics in a DoD solicitation lies with the Integrated Review Team, which contains representatives from each of the awarding components. Topics are reviewed initially at DDR&E and are then returned to the agencies for correction of minor flaws, for revision and resubmission, or as discards.

DoD has also made considerable efforts to improve the topic selection process. In the late 1990s, DoD determined that topics were not being linked closely enough to the users of SBIR technologies in the acquisition community, and a conscious effort was made to ensure that more topics were “owned” by that community (with considerable success—see DoD report, Chapter 5: Program Management, for details²⁸).

4.3.2. Program Outcomes and Agency Mission

In contrast to the discussion above, the program outcomes for agency mission are both more difficult to assess and also more specific to individual agencies. All of the methodological difficulties in assessing outcomes discussed at the beginning of this chapter apply here; moreover, (unlike commercialization) there are no obvious and widely understood benchmarks that apply across agencies.

All agencies maintain a list of “success stories,” describing SBIR awards that meet congressional goals. Some of these are focused on agency mission. However, the stories themselves, while illustrative of the power of the program to help develop new technologies, are of variable quality. Some agencies, such as NIH, use success stories written by the company in question without validating them. Even agencies that take a more systematic approach to case studies do not appear to have clear and transparent criteria for determining what should count as a “success story.”

Each agency has found different ways to explore and describe outcomes related to agency mission. As a result, our overview must reflect these agency differences. Below, we provide a brief summary of agency approaches to identifying and measuring ways in which the SBIR program supports the agency mission.

4.3.2.1. NIH

For NIH, the primary mission is the development of fundamental knowledge and its application for improving health.

The NIH mission is science in pursuit of fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability.

SOURCE: National Institutes of Health, <http://www.nih.gov/about/ndex.html#mission>.

A more detailed assessment of outcomes related to these objectives can be found in Chapter 3 of the NIH Report.³¹ Data on patents and peer-reviewed publications resulting from awards can be used as an indicator of the development of fundamental knowledge. Commercialization measures are another possible indicator of the transfer of knowledge, as commerce marks a transaction that reflects that transfer.

Still, more direct measures would be useful. NIH has tried to develop these in several ways. Like other agencies, NIH maintains a Web page filled with “success stories.” Unfortunately, these are entirely self-posted by companies, and reflect no selection or even verification by staff.

The NIH SBIR program has also led the way at NIH in developing metrics through the implementation of a recipient survey, with the first survey being deployed in late 2003. In this survey, the agency has sought to identify the populations targeted by SBIR projects. Figure 4-9 shows the distribution of projects by sector, for both projects that have already reached the market and those still being commercialized.³²

FIGURE 4-9

Distribution of NIH projects, by type of affected population. SOURCE: National Institutes of Health, National Survey to Evaluate the NIH SBIR Program: Final Report, July 2003.

The NIH survey also sought to quantify the number of people affected by a technology, asking respondents to place their project within ranges of affected populations by size. Such information, if accurate, would be useful. Unfortunately, respondents have wide latitude in answering, and are given no guidelines that might help to explain how to structure their answers. The responses are essentially guesses—perhaps biased guesses, as respondents may have a tendency to overestimate their projects’ importance.

Still, the data categories show the wide variety of mission-related areas into which SBIR projects can be categorized—and all clearly fall within the broad mission definition of NIH. Responses can also indicate the distribution of projects between mission areas and can be used as a proxy for the degree of NIH interest in these different areas.

4.3.2.2. DoD

The DoD SBIR program’s primary goal is the provision of technologies that are employed as part of defense systems being developed, acquired, or maintained to meet DoD mission needs. In effect, that acquisition stream defines, for operational purposes, agency needs, and the extent to which SBIR is, in fact, part of the acquisition process has become an important indicator of support for agency mission.

The mission of the Department of Defense is to provide the military forces needed to deter war and to protect the security of our country.

SOURCE: Department of Defense, <http://www.defenselink.mil/admin/about.html>.

This was not always the case. As discussed in the DoD volume, the program’s objectives have evolved since the late 1990s. Some elements of the management have recently focused on finding ways to more closely align SBIR projects and programs with the acquisition process of the Services.

Much of the discussion that follows will focus on acquisitions and on Phase III at DoD. However, it is still important to remember that, even at DoD, not all outcomes that support the mission result in acquisitions and Phase III. For example, agency staff have indicated that SBIR can be an important way of assessing technologies that in the end do not pan out. The awards thus act as a low-cost probe of the technological frontier. So, even when no acquisition occurs, SBIR can still provide valuable support for the agency mission in terms of information on technological dead ends, promising technological options, or use resulting from the award itself.

Still, even with these multiple functions, as participants at the NRC’s Phase III Symposium affirmed, Phase III and acquisitions are regarded by DoD as the core focus of the SBIR program, and the key indicator for measuring success in supporting agency mission. Many speakers at the NRC’s Phase III Symposium made clear that the take-up of SBIR-funded technologies into the DoD acquisitions program was the benchmark against which DoD’s SBIR program should be judged.

DoD has made a more conscious effort to measure SBIR impacts on agency performance than any of the other agencies under study. Because DoD is a procurement agency, there is also one clear set of indicators which, if properly captured, could provide critical benchmarks and feedback to the agency.

DoD has two tools for measuring follow-on Phase III contracts. The first is the DD350 contracting forms, which is supposed to capture whether a contract is a follow-on to an SBIR award. However, only well-trained contracting officers who understand SBIR are likely to correctly fill out the form. Internal assessment of the DD350 by DoD suggests that not all contracting officers use the form correctly. Consequently, at many DoD components, SBIR follow-on contracts are only erratically reported, if at all.³³ The other tool for measuring follow-on Phase III contracts is the DoD SBIR/STTR Commercialization Database. This database is used to calculate firms’ Commercialization Achievement Index and to generate the Company Commercialization Reports that are packaged with proposals to evaluators.

The Navy has recently made efforts—including the investment of external program dollars—to improve the quality of DD350 data. This effort is reflected in recent DD350 results, which show Navy as accounting for more than 70 percent of all DoD Phase III awards.³⁴

The DD350 results do show that the amount of Phase III contracts generated have been climbing steadily in recent years, particularly at the Navy (see Figure 4-10).

FIGURE 4-10

Phase III awards, annual totals, 1999-2005. SOURCE: DoD SBIR/STTR Commercialization Database, provided by John Williams, U.S. Navy, April 7, 2005.

These data also show that the amount of Phase III contracts being identified is greater than the amount of funding expended on SBIR, once commercialization has been appropriately lagged.

A fuller discussion of DoD’s efforts regarding Phase III can be found in Chapter 5 of the study of SBIR at DoD.³⁵ Leaving complexities aside, however, it is possible to draw the following general conclusions:

FIGURE 4-11

Reported commercializations versus SBIR budget.

SOURCE: Michael Caccuitto, DoD SBIR/STTR Program Administrator, Presentation to SBTC “SBIR in Rapid Transition” Conference, Washington, DC, September 27, 2006.

• DoD is increasingly making an effort to measure support for agency mission.
• Part of this assessment is increasingly quantitative, with the Navy leading the way in utilizing data from the DD350 forms. Nonetheless, there are barriers and difficulties that tend to reduce the number and amount of Phase III contracts counted in the DD350 tracking forms.
• The demonstration effect of the Navy program and growing awareness of SBIR’s potential has increased senior management’s awareness of SBIR’s contributions to the Defense mission.³⁶
• The growing interest and dedicated management attention by Prime contractors is enhancing the potential for insertion of SBIR outputs into the acquisition process.
• The reluctance of other agencies to provide management funding at the level of the Navy’s effort remains a constraint on maximizing the return on the nation’s SBIR investment.

In short, recent improvements in DoD activities related to Phase III do suggest a slow but significant change in attitude toward SBIR within DoD. SBIR is seen less as a tax imposed by Congress on otherwise useful R&D activities, and more as an opportunity to bring innovative ideas and products to meet mission objectives.

4.3.2.3. DoE

DoE has, to a considerable extent, relied on process and procedures to ensure that its mission is being supported by the SBIR program. The agency has developed topic and award selection procedures that ensure that the primary driver of the program will be R&D managers within the agency, rather than either external peer reviewers or SBIR program managers. The latter play a minimal role in topic and award selection.

Mission

Discovering the solutions to power and secure America’s future

Vision

The Department’s vision is to achieve results in our lifetime ensuring: Energy Security; Nuclear Security; Science-Driven Technology Revolutions; and One Department of Energy—Keeping our Commitments.

SOURCE: Department of Energy, <http://www.cfo.doe.gov/strategicplan/mission.htm>.

In FY2005, this process resulted in 49 technical topics, from within the 12 program areas (see Table 4-6).

TABLE 4-6

DoE Phase I Awards by Program Area.

There does not appear to be a systematic effort under way at DoE to determine whether the SBIR program supports agency mission. As with all other agencies, to the extent that agency mission involves the development of technical knowledge, there are indicators for whether SBIR companies are producing this knowledge.

4.3.2.5. NASA

NRC used a unique approach to address the question of support for NASA’s agency mission. NRC surveyed agency technical managers (COTRs), who are in charge of the research areas within which SBIR awards are made. The survey sought to measure the quality of the research from the perspective of technical staff who managed both SBIR and non-SBIR programs.

NASA’s mission is to pioneer the future in space exploration, scientific discovery, and aeronautics research.

SOURCE: National Aeronautics and Space Administration, <http://www.nasa.gov/about/highlights/what_does_nasa_do.html>.

The research found that the COTRs believed that 68 percent of surveyed projects generated useful information and found that results from 58 percent of the projects were sufficiently positive to encourage the COTR to seek out additional funding, within or outside the SBIR program.

It is especially encouraging that over 30 percent of projects were sufficiently positive that technical managers sought non-SBIR funding within NASA for further development.

Agency mission at NASA can also be measured in terms of agency take-up of SBIR-funded technologies. This may be especially important at NASA, as the commercial market for space-related technologies is likely to be small. NASA does maintain a set of success stories, but Phase III activities do not appear to be tracked in any coordinated fashion.

FIGURE 4-12

NASA staff perspectives on SBIR awards.

SOURCE: NRC Project Manager Survey.

4.4.1. Support for Woman- and Minority-owned Firms

Support for women and disadvantaged persons is one of the four primary congressional objectives for the SBIR program. Unfortunately, there are significant concerns regarding the collection of data related to the participation of woman- and minority-owned firms in the program. More importantly, the data that does exist raises questions about whether agencies adequately meet the needs of such firms.

4.4.1.2. Award Patterns

Of the five agencies studied by the NRC, one—NASA—has no apparent issues in relation to awards to woman- and minority-owned firms. That is to say, their pattern of awards matches those for other firms, and their share of awards is in line with the average for all agencies, and shows no obvious negative trends in any area.

This is not the case for the remaining four agencies. At DoD, there has been a substantial decline in the award shares of minority-owned firms. The share of awards to woman-owned firms has been relatively low at NSF, and the application success rates for woman-owned firms have been lower than those for other businesses. At DoE, success rates for both woman- and minority-owned firms have been lower than for other small businesses. At NIH, issues focus on award shares for minority-owned firms, and on the discordance between the award share to woman-owned firms and the number of female scientists and engineers working in the life sciences. And, as noted the lack of quality data suggest that this issue is not adequately monitored or analyzed, reflecting a need for greater management attention.

DoD

While Phase I awards to woman-owned firms have continued to increase as a percentage of all Phase I awards, the percentage of Phase I awards being made to minority-owned firms has declined since the mid-1990s. The percentage fell below 10 percent for the first time in 2004, and is down by a third since the early 1990s.

The absence of reliable data on the demographics of applications makes it impossible at this point to determine why minority-owned businesses are getting a declining share of Phase I awards. We do not know whether minority-owned business applications are down, whether the success rate of those applications has fallen, or both.

NSF

At NSF, the existence of good applications data makes it relatively easy to determine why woman-owned firms have been doing less well in recent years. Woman-owned businesses have been less successful in getting applications approved than have all applicants in every year since 1994. In half the years, the success rate of woman-owned businesses was less than 70 percent the rate of all applicants. Over the period 1994-2005, woman-owned businesses accounted for 12 percent of all Phase I applications but received less than 10 percent of all Phase I awards.

FIGURE 4-13

Minority-owned business shares of Phase I awards at DoD, 1992-2005.

SOURCE: Department of Defense Awards Database.

FIGURE 4-14

NSF: Comparative success rates for woman-owned and for all applicants in having their Phase I applications approved, 1994-2005.

NOTE: This is a correction of the prepublication version released on July 27, 2007.

SOURCE: Developed from data provided by the NSF SBIR program.

FIGURE 4-15

NSF: Comparative Phase II success rates for woman-owned and for all applicants, 1995-2005.

NOTE: This is a correction of the prepublication version released on July 27, 2007.

SOURCE: Developed from data provided by the NSF SBIR program.

Low success rates occur during Phase II as well. In all except three of the ten years from 1995 through 2005, the success rate of woman-owned businesses fell below that of all businesses in having their Phase II applications funded. Woman-owned businesses contributed 9 percent of all Phase II applications submitted from 1995 to 2003 (203/2,299) and received 7.5 percent of all Phase II grants (79/1,059). From 2002 to 2003, the success rate of woman-owned businesses in getting Phase II grants was particularly low but recently has recovered significantly.

DoE

At DoE, success rates are also an area of interest. Woman-owned businesses accounted for almost 8 percent of all DoE Phase I awards during 1992-2003, and for 10.5 percent of Phase I applications. During the same period, woman-owned businesses accounted for 7.8 percent of Phase II applications in 1995-2003 and for 6.7 percent of actual Phase II awards.

Data on minority-owned firms was similar. They submitted 16.5 percent of all Phase I applications in 1992-2003 and received 13.2 percent of all Phase I awards. Between 1995 and 2003, they accounted for 13.7 percent of Phase II applications and 11.8 percent of Phase II awards.³⁸

DoE is the only agency where minority-owned firms apply more and receive more awards than woman-owned firms.

NASA

The relatively steady trends for both Phase I and Phase II awards at NASA for both woman- and minority-owned businesses are shown below (see Figures 4-16 and 4-17).

FIGURE 4-16

Phase I awards at NASA: woman- and minority-owned businesses’ share of all awards. NOTE: This is a correction of the prepublication version released on July 27, 2007.

FIGURE 4-17

Phase II awards at NASA: woman- and minority-owned businesses’ share of all awards. NOTE: This is a correction of the prepublication version released on July 27, 2007.

The absence of detailed applications data for woman- and minority-owned businesses means that we currently are not able to determine whether these trends are the result of a faster increase in these firms’ number of proposals than other small businesses, improved success rates, or a combination of both.

4.4.2. Small Business Support

At one level, the SBIR program obviously provides support for small business, in that it gives funding only to businesses with no more than 500 employees—the SBA definition of a small business. However, it has been less clear whether SBIR has provided additional support for small business, or simply aggregates existing small business research dollars under the program’s umbrella.

The most direct way to address the question is to review the amount of R&D funding going to small business at the agency as a whole, and then compare that with trends in SBIR funding. Unfortunately, this comparison is publicly available only for one agency, NIH, which publishes separate data on small business shares of research funding.

NIH data (Figure 4-18) show that the share of all NIH small business funding being disbursed through the SBIR program has fallen steadily since soon after the 1983 inception of the NIH program. After peaking at 90 percent of all small business research funding in the mid-1980s, SBIR’s share fell steadily to about 72 percent, before falling further in 2004 (the most recently available data).

FIGURE 4-18

SBIR share of small business research funding at NIH SOURCE: National Institutes of Health Awards database.

At NIH, the rapid growth of SBIR in recent years has supplemented, not supplanted, small business funding through other mechanisms at NIH. Increasing amounts—and shares—of small business research funding are available outside the SBIR program.

4.4.3. Project-level Impacts

One way of measuring SBIR’s impact is to ask awardees whether their projects would have been implemented without SBIR program funding. Data from the NRC Phase II Survey shown in Figure 4-19 strongly suggest that SBIR provides funding that plays a determinant role to most of the projects that receive it.

FIGURE 4-19

Would the project proceed without SBIR funding? (Percent of respondents.) SOURCE: NRC Phase II Survey, Question 13.

According to the respondents, more than 70 percent of projects would likely not have proceeded at all without SBIR. This finding reflects the known difficul ties in funding high-risk early-stage research in all scientific fields. SBIR seems to provide critical funding necessary to fund many early stage projects.

Respondents also indicated that many of the 13 percent of projects that were “definitely” or “likely” to have continued in the absence of SBIR funding would have had significant delays and other changes. More than half of these respondents (54 percent) noted that the project’s scope would have been narrower. Sixty-two percent of projects that would have continued would have been delayed, and 47 percent expected the delay would have been at least 12 months.³⁹

4.4.4. SBIR Impacts Different Types of Companies in Different Ways

Professor Irwin Feller has proposed a typology of companies supported by SBIR funding, capturing the critical differences in company capabilities and aspirations. This section describes, using both data and case studies, how these five different kinds of companies (or different stages of company development) have been supported by the NIH SBIR program.

Feller’s typology differentiates between five types of SBIR-supported company:

1. Start-up Firm. This is a new firm, typically without marketable products, and usually with minimal funding and limited personnel resources.
2. R&D Contractor. As described by Cramer, these firms make a strategic choice to specialize in the performance of R&D rather than in marketing products or services.⁴⁰
3. Technology Firm. These firms have developed a core technology, which is then deployed into products and services.
4. Scientific Firm. These businesses are described by Cramer as “firms [that] are generally small and were founded by scientists to explore whether a particular research area can generate ideas or products that might attract investors.”⁴¹
5. Transformational Firm. These companies start out as highly (or partially) dependent on SBIR, or on other government R&D contracts, which they use to develop a product that turns out to have considerable commercial value. This leads the company to become a production-oriented commercial vendor, with a concomitant decrease in the role of SBIR on the firm’s progression.

All of these firm types have been identified through case studies as benefiting from SBIR awards. However, SBIR impacts on company formation are an especially important component of the program’s overall support for companies; in addition, the role of R&D contractor companies has been a matter of controversy within the program for some time.

4.4.4.1. Start-ups

Responses to the NRC Firm Survey indicate that just over 20 percent of firms were founded entirely or in part as a result of SBIR awards. These data are supported by a number of the cases examined for this study. One of them is summarized in Box 4-5.⁴²

BOX 4-5

Company Formation: The Case of Sociometrics, Inc. Sociometrics is a successful winner of SBIR awards that has also successfully commercialized some of its SBIR-generated technologies. Over the past twenty years, it has received more than 15 Phase II awards, (more...)

While supporting company formation is not a direct objective of the program, it is easily construed as part of the mandate to support small businesses, and the program does, in fact, meet this important objective.

4.4.4.2. R&D Contractors

R&D contractor-type companies have been the subject of pointed criticism from some observers of the SBIR program. These critics have noted that some of these companies receive substantial SBIR program funds, but produce little in the nature of commercial goods and services. There is also minimal “take-up” of their R&D into the acquisition streams of the procurement agencies. These companies have been derogatively referred to as SBIR “mills,” living off SBIR awards.

This issue is discussed in more detail in the section in Chapter 2 on multiple-award winners. Here, however, we should simply note that this criticism often misses important changes at these companies. The last point has more general application: reliance on SBIR awards tends to decline as the size of the company grows, and most companies that survive do grow. This observation is supported by data from the NRC Phase II Survey.

Table 4-8 shows, for NIH awardees, how responding firms’ SBIR focus compares with their revenue at the time of the survey (2005). The data show that as companies get larger, their reliance on SBIR revenues tends to decline. Of the 36 NIH respondents with at least $5 million in revenues, 30 (78.9 percent) reported no more than a 10 percent focus on SBIR. Conversely, of the 102 firms reporting at least 80 percent focus on SBIR, 100 reported annual firm revenues of no more than $1 million. The data also show that there are a number of companies with revenues of $1-5 million where SBIR accounts for more than half of the revenues.

TABLE 4-8

NIH Phase II SBIR Awards by Overall Company Revenue and Percentage Dependence on SBIR as a Source of Revenue.

4.4.5. Growth Effects

No existing data sets measure the effect of SBIR on company growth. However, NRC survey respondents did provide their own estimates of SBIR impacts on their companies’ growth.

Almost 55 percent of respondents indicated that more than half of the growth experienced by their firm was directly attributable to SBIR. This is evidence of the powerful impact of SBIR on the future development of firms winning SBIR awards.

FIGURE 4-20

SBIR impacts on company growth: Percent of company growth attributable to SBIR awards.

SOURCE: NRC Firm Survey.

4.4.6. Conclusions

SBIR supports small high technology businesses at a time when other sources of financial support are especially difficult to find. Businesses use these funds for a variety of purposes, in pursuit of several distinct strategies, as captured in Box 4-6.

BOX 4-6

SBIR as an Enabler and Lifeline for Some High-Tech Companies. Faraday Technologies, Inc. The SBIR program enabled the company to undertake research that otherwise it would not have done. It sped the development of proofs of concept and pilot-scale prototypes, (more...)

4.5.1. The Incidence of Multiple-Award Winners

The absence of a high quality unified database of award winners makes it difficult to track MAWs. Because there is no single ID for SBIR firms, MAWs must be tracked using firm names, and this method is not reliable: not only do firms change names, even the smallest change in the spelling of a firm name on the application cover sheet will create a second record for that firm.

4.5.2. Commercialization and Multiple-Award Winners

While the DoD commercialization database does not contain information on all companies or all awards,⁴³ it does provide the best data on overall outcomes from awards to multiple SBIR-award winners, partly because it is specifically designed to do so. DoD has been concerned to ensure that companies winning more than four or five awards (the number changed in 2005) are generating results that support DoD’s mission.⁴⁴

The DoD data covers firms and awards from all agencies, reported within the DoD system (by firms that apply for DoD awards). It is therefore incomplete. However, it does show both the fact that a relatively small number of companies account for a larger share of awards, and also that there are now very many firms that win a great many awards.

The top 1 percent of winners—27 firms—accounted for 16.8 percent of all recorded Phase II projects. The top 7 percent of award winners—200 firms—accounted for 46.5 percent of all awards.

Of course, these data alone do not determine whether any of these firms are “mills”—whether they do not commercialize, and whether they are SBIR-dependent for revenues.

The first question can be answered, at least in aggregate. As noted, data from the DoD commercialization database indicates that the largest award winners are in fact—on average—stronger commercializers than firms with few awards.

The 2,920 firms reporting through the commercialization database reported a total of $20.3 billion in commercial outcomes, including $13.2 billion in sales. The five companies with more than 124 projects generated approximately 60 percent greater commercial returns on average than did the projects with fewer than 15 awards, and accounted for $1.4 billion in sales.

The 7 percent of firms with the highest number of awards—which accounted for 46.5 percent of awards, accounted for 60 percent of the commercial outcomes. This does provide a rebuttal to the claim that multiple winners are simply mills: At the most basic level, the data show that the multiple winners commercialize more than firms receiving fewer awards.

Thus the criticism of MAWs seems in general to be misplaced. It results from an overly negative reading of limited data focused too tightly on raw commercialization data, and perhaps on assessments of the program in its first years of operation.⁴⁵

A fuller assessment of the role of multiple winners reveals multiple dimensions, and multiple contributions:

• Active Commercialization. Aggregate data from the DoD commercialization database indicates that the companies winning the most awards are, on average, more successful commercializers than those winning fewer awards.
  • While data from this source are not comprehensive, they do cover the vast majority of MAWs—at least at DoD—and the data indicate that, on average, firms with the largest number of awards commercialize as much or more than do all other groups of awardees; in the aggregate, there is failure of multiple-award winners to commercialize.
  • Case studies also show that some of the biggest award winners have successfully commercialized, and have also met the needs of sponsoring agencies in other ways.

• Declining SBIR Revenues. For some MAWs at least, even though they continue to win a considerable number of awards, the contribution of SBIR to overall revenues has declined: At the commonly cited “mill” firm Radiation Monitoring Devices, for example, SBIR now only accounts for 16 percent of total firm revenues.
• Graduation. Some of the biggest Phase II winners have graduated from the program either by growing beyond the 500 employee limit or by being acquired. Foster-Miller, for example, was bought by a foreign-owned firm. Legislating to solve a problem with companies that are, in any event, no longer eligible seems inappropriate.
• Agency Research. Contract research can be valuable in and of itself. Agency staff indicate that SBIR fills multiple needs, many of which do not show up in sales data. For example, some agency staff have noted that they use SBIR awards to conduct low-cost probes of the technological frontier. Awards are conducted on time and on budget, and can effectively test technical hypotheses, potentially saving extensive time and resources later.
• Spin-off Companies. Some MAWs spin off companies to commercialize innovations. Creating new firms is a valuable impact and a valuable contribution of the program.
• Rapid Response. MAWs have provided the highly efficient and flexible capabilities needed to solve pressing problems rapidly. For example, Foster Miller Inc. responded to needs in Iraq by developing and the manufacturing add-on armor for Humvees.⁴⁶

All these points suggest that while there have been companies that depend on SBIR as their primary source of revenue for a considerable period of time, and there are some who fail to develop commercial results, such behavior is limited and does not constitute a significant impediment to SBIR program goals.

There is one final point to consider. Given the fact that SBIR awards meet multiple agency needs and multiple Congressional objectives, it is difficult to see how the program might be enhanced by the imposition of an arbitrary limit on the number of applications per year, as is currently the case at NSF. The evidence—outlined above—supports the conclusion that there is no multiple winner problem per se, although there may be “problem,” i.e., underperforming, firms—a judgment the agency management must make. To the extent agencies continue to see issues in this area, we would strongly suggest that they consider adopting some version of the DoD “enhanced surveillance” model, in which multiple winners are subject to enhanced scrutiny in the context of the award process. The exclusion model used by NSF seems simply inappropriate.

4.5.3. The Incidence of New Entrants

Just as analysis of multiple winners provides a view of award concentration, analysis of new entrants provides a view of award dispersion. The data unequivocally show that the agency SBIR programs are open to significant numbers of new entrants.

4.6.1. Patents

According to the Small Business Administration, small businesses produce 13 to 14 times more patents per employee than do large patenting firms. These patents are twice as likely as are large firm patents to be among the one percent most cited.⁴⁸

With respect to SBIR, the NRC survey data indicate that about 30 percent of respondents received patents related to their SBIR research. About two-thirds of projects generated at least one patent application, and as with most other SBIR metrics, the distribution of outcomes is highly skewed.

The data show that three companies account for almost 20 percent of all patent applications related to the surveyed SBIR award. However, they were not especially successful, as no company reported more than 20 successful patents granted. Overall, 2.2 percent of the responding companies applied for more than five patents, and 1.3 percent received more than five.

More detailed case study analysis also indicates the importance of intellectual property to firm strategies. At NSF, a number of the case studies showed this clearly.

4.6.2. Scientific Publications

The NRC Phase II Survey determined that slightly more than half of the respondents had published at least one related scientific paper (45.4 percent). About one-third of those with publications had published only a single paper, but one company had published 165 papers on the basis of its SBIR research, and several others had published at least 50 papers.

These data fit well with case studies and interviews, which suggested that SBIR companies are proud of the quality of their research. Publications are featured prominently on many Web sites, and companies like Advanced Brain Monitoring, SAM Technologies, and Polymer Research all made the point during interviews that their work was of the highest technical quality as measured in the single measure that counts most in the scientific community—peer-reviewed publication.

Publications therefore fill two important roles in the study of SBIR programs:

• First, they provide an indication of the quality of the research being conducted with program funds. In this case, more than half of the funded projects were of sufficient value to generate at least one peer-reviewed publication.
• Second, publications are themselves the primary mechanism thorough which knowledge is transmitted within the scientific community. The existence of the articles based on SBIR projects is therefore direct evidence that the results of these projects are being disseminated widely, which in turn means that the congressional mandate to support the creation and dissemination of scientific knowledge is being met.

We note that no agency seems to have in place evaluation programs for determining whether similar knowledge effects are being generated at the same or at a different rate outside the SBIR program. Likewise, there is insufficient tracking to determine citation rates for these articles.

4.6.3. SBIR and Universities

According to agency and congressional staff, one of the implicit, though not formal objectives, of the SBIR program is that it supports the transfer of knowledge from universities to the commercial marketplace.

This transfer can happen in many different ways. SBIR funding:

• Helps university scientists form companies;
• Enables small firms to use university faculty and to employ graduate students as specialized consultants;
• Permits the use of university laboratory facilities; and
• Encourages other less formal types of collaboration.

Over a third (36.5 percent) of all NRC Phase II Survey respondents indicated that there had been involvement by university faculty, graduate students, and/or a university itself in developed technologies. Based on the responses of all survey respondents, this involvement took a number of forms (see Table 4-13).

TABLE 4-13

University Involvement in SBIR Projects.

The wide range of roles played by university staff and students underscores the multiple ways in which SBIR projects enhance the knowledge base of the nation. Involvement in these projects provides different opportunities for university staff than those available within academia.

Many of the companies interviewed for this study noted that their university connections were extremely important. Radiation Monitoring Devices said that they spent more than $1 million annually on contracted research at universities under their SBIR awards.⁴⁹ Advanced Targeting Systems, in San Diego, has forged an extended and successful research partnership with a senior scientist at the University of Utah, which they described as “critical to the development and testing of their products.”⁵⁰

These stories and the many others like them that we encountered suggest that the flow of information and funding between small businesses and universities working within the SBIR framework is not simple or unidirectional. For example, the constant flow of feedback, testing, and insights between university researchers and Advanced Targeting Systems staff helped to move the company forward toward product deployment into new research areas while providing real-life problems for research to address.

What is now clear from this research—and could certainly be the subject of further analysis—is the extent to which universities themselves see SBIR as a mechanism for technology transfer, commercialization, and additional funding for university researchers.