5.2.1. Acquisition-oriented Approaches

Acquisition-oriented approaches are designed to align long range research with the likely needs of specific agencies and programs. How well this works depends to a great degree on the specific topic development mechanisms used at each agency.

5.2.2. Management-oriented Approaches to Topic Utilization

Management-oriented approaches differ from the acquisition-driven programs in that they do not evaluate applicants based on the government agency’s acquisitions needs.

Instead, there are management priorities. The two agencies using this approach differ substantially in the origins of these priorities. At NSF, SBIR funding decisions are driven by the technical decisions of the major funding directorates. Each directorate provides one annual list of topics, and their objective is explicitly to direct SBIR funding into targeted research areas. DoE’s priorities are described in the case study below.

5.2.3. Investigator-driven Approaches

NIH is proud of its investigator-driven approach. The agency believes its approach is the best way to fund the best science, because it substitutes the decisions of applicants and peer reviewers for the views of agency or program staff.

A small but growing percentage of NIH SBIR funding, however, goes to Program Announcements (PAs) and Requests for Applications (RFAs), which are Institute-driven requests for research proposals on specific subjects, covering the remainder. Discussions with agency staff suggest that this percentage may increase, perhaps substantially.⁸

5.2.4. Topics: Conclusions

There are some obvious advantages to an investigator-driven approach:

• Applications are likely to respond to current concerns more rapidly (given the long lags involved in topics developed with other mechanisms).
• Multiple deadlines are easier because an official solicitation does not have to be adjusted.
• Better science may be attracted, as promising and truly innovative new technologies are not excluded.

Acquisition- and management-driven models have their own advantages:

• Agencies can much more easily impose their own research agendas.
• Agencies can better ensure that SBIR research is aligned with eventual agency acquisition needs.
• Narrow technical “windows” mean that the agency can limit the technologies being addressed by a particular solicitation.
• Narrower topic options mean fewer applications, limiting the amount staff and reviewer time required.

The case for better science may be the most important point here. By definition, if topics are designed to sharply limit/focus research on particular problems, or in some cases on particular technologies, they are excluding other kinds of research. And, simple mathematics would suggest, if possible applicants are excluded a priori, the average quality of successful applications is likely to fall.

This suggests the following conclusions:

• The acquisition-oriented model seems appropriate at the agencies where eventual technology acquisition by the agency is the primary objective.
• The management-oriented model seems much less immediately defensible.
  • It is difficult to see how the narrowing of potential topics at NSF best serves the interest of science: in conjunction with the annual funding cycle, this ensures that excellent science may have to wait for several years before its number comes up in the NSF topics lottery.
  • It is even more difficult to defend the narrowing of selection areas on the basis of administrative convenience, as at DoE. Limited resources mean that all agencies limit the number of SBIR awards that are made, but it is hard to understand how doing so through a deliberate decision to reduce the scope of possible applications makes much sense.
• The NIH investigator-driven model seems to be working effectively for NIH.
• Hybrid models are worth further assessment. NIH appears to be evolving toward a hybrid model, as the share of awards going to RFAs and PAs is growing. This suggests possible options with which other agencies could experiment. For example, DoD components might reserve a small percentage of funding for “open” topics that are essentially investigator-driven.

It is worth noting that in other countries, hybrid models are used. For example, in Sweden, approximately half of Vinnova’s⁹ research funding is distributed to agency programs focused on the research needs of specific industries. The other half is allocated to proposals for research initiated by companies.¹⁰

5.3.1. Introduction

As the program has become larger and more important to early-stage funding for high-tech companies, outreach activities by the programs have increasingly come to be complemented by outreach initiatives at the state level.

At the different agencies, outreach programs seem to share three key objectives:

• Attracting new applicants.
• Ensuring geographical diversity in applications by attracting new applicants from under-represented regions.
• Expanding opportunities for disadvantaged and woman-owned companies.

These objectives are implicit, rather than explicit, in the outreach activities of all the programs. With the growth in electronic access, it is unclear whether greater outreach efforts could be cost-effective.

5.3.2. Outreach Mechanisms

At the program level, all the agencies conduct outreach activities, though those efforts vary in kind and degree. Important types of outreach activities include:

• National SBIR Conferences, which twice a year bring together representatives from all of the agencies, often at locations far from the biggest R&D hubs (e.g., the Spring 2005 national conference took place in Omaha, Nebraska).
• Agency-specific Conferences, usually held annually, which focus on awardees from specific agencies (e.g., the NIH conference, which is usually held in Bethesda, Maryland).
• Phase III Conferences, focused on bringing together awardees, potential partners, such as defense industry prime contractors, and potential investors (e.g., the Navy Opportunity Forum, held annually in the Washington, DC area).
• The SWIFT Bus Tour, which makes an annual swing through several “under-represented” states, with stops at numerous cities along the way. Participants usually include some or all of the agency program managers/directors/coordinators.¹¹
• Web Sites and Listservs are maintained by all of the agencies. These Web sites contain application and support information. Some, such as the DoD’s site, are elaborate and comprehensive, and most allow users to sign up for a news listserv.
• Agency Publications are used to spread information about the SBIR program. NASA, for example, has a variety of print and electronic publications devoted to technology and research promotion. The other departments use this publicity vehicle to a lesser degree.
• Other, non-SBIR-focused Publications are sometimes used by the agen cies. For example, DoD publishes solicitation pre-publication announcements in its Commerce Daily.
• Demographic-focused Outreach. Agencies acknowledge the need to ensure that woman- and minority-owned businesses know about SBIR and are attracted to apply. They have organized a series of conferences to encourage participation in the program by woman- and minority-owed business, often run in conjunction with other groups. For example, the DoD Southeastern Small Business Council held a conference for Woman-owned Small Businesses in Tampa, Florida, in June 1999.

In general, these activities are the responsibility of the SBIR Program Coordinator at the agency, rather than other agency staff. They are also funded out of general agency funds, as the governing SBIR legislation does not permit programs to use SBIR funds for purposes other than awards (and a limited amount of commercialization assistance. See below.).

Generally, agencies do not provide much funding for outreach, though there are pronounced differences between the agencies. NIH, for example, has developed a popular annual SBIR conference, whereas DoE has no funds for such an initiative. Within DoD, the Navy has provided significant extra operating funds for its SBIR program, which has therefore been able to take on outreach and other initiatives not open to the programs at other DoD components.

5.3.3. Outreach Outcomes

5.3.3.1. New Winners

As SBIR programs have become larger, with longer track records and better online support, it is increasingly less likely that potential candidate companies will remain unaware of the program, or that such companies will have difficulty understanding how to apply.

Still, new applicants remain an important component of the applicant pool (see Figure 5-1, for example). This may suggest that the current outreach activities are having the intended effect. Much learning about new opportunities such as SBIR is conveyed directly. Another interpretation of the data is that outreach activities have become increasingly unnecessary in a world where a wide range of economic development institutions at the state and local level automatically point small high-tech companies toward SBIR as a funding source, and where public information about SBIR is fairly widespread and online.

FIGURE 5-1

Phase I new winners at NSF (previously nonwinners at NSF). SOURCE: National Science Foundation.

Anecdotal evidence from the SWIFT bus tour participants suggests that outreach activities do encourage new firms to enter the program. The program coordinator for NIH has noted that there is usually an upturn in applications from visited states after the tour, especially from states which have had relatively few applicants in the past.¹²

5.3.4. Conclusions—Outreach

Information for potential applicants falls into two areas: (1) basic program information for potential applicants who have not heard of the program or have not yet applied and won awards; and (2) detailed technical information about specific solicitations and opportunities for those familiar with the program.

Interviews with agency staff and SBIR awardees indicate that the agencies have substantially upgraded their SBIR online information services in recent years. The Web has become the primary source of information for both existing awardees and potential new applicants.

While some of the Web sites are still not especially easy to navigate or utilize, collectively they represent a vast improvement in outreach capacity for the agencies. Moreover, online services help to ensure that potential applicants get up-to-date information.

For those with little information about the program, the states have increasingly come to play an important information-providing role. Almost all states have a technology-based economic development agency (TBED) of some kind. Many have TBEDs that are well-known and effective organizations. These or ganizations are aware of SBIR. They help to organize local and regional conferences with the SBIR agencies, they attend other SBIR events, and they are often in direct contact with agency staff.

Some TBEDs, such as the Innovation Partnership in Philadelphia, include SBIR as a key component. They center many of their activities around SBIR by helping local companies develop winning applications and integrate SBIR into a wider system for economic development. For example, the Innovation Philadelphia has a multiple funding model for early stage financing, which includes a role for SBIR.¹⁴

Overall, most companies with some reasonable likelihood of success in applying to the SBIR program have heard about SBIR and can find out how to fill in the appropriate application forms. To the extent that they want help in the process, there is a small cottage industry of SBIR consultants eager to provide that service.

Thus, it appears that the general outreach function historically fulfilled by the SBIR Agency Coordinators/Program Managers may now be changing toward a more nuanced and targeted role. Outreach can focus on enhancing opportunities for under-served groups and under-served states, or on specific aspects of the SBIR program such as the transition to the market after Phase II (e.g., the July 2005 DoD “National Phase II Conference—Beyond Phase II: Ready for Transition” held in San Diego), while relying on the Web and other mechanisms to meet the general demand for information.

5.4.1. Introduction

SBA Guidelines require that an agency should use the same selection process for its SBIR program as it does for its other R&D programs; as award selection processes differ across agencies, so do selection procedures for SBIR.

The selection process is critical for the long-run success of any government award program. For SBIR, the selection process is complicated by the fact that the program serves many masters and is aimed at many objectives. A successful selection process must therefore meet a number of quite distinct criteria. Discussions with agency staff and award winners, and with the other stakeholders, suggest that the following are key criteria for judging the quality of the SBIR selection processes:¹⁵

• Fair. Award programs must be seen to be fair, and the selection process is a key component in establishing fairness. Despite numerous complaints about details of the NIH selection process, there is uniform acceptance among stakeholders that, in general, the process is fair.
• Open. A successful SBIR program must be open to new applicants. Agencies generally score high on this, with a third or more of Phase I awards at most agencies going to new winners.
• Efficient. The selection process must be efficient, making the best of use of the time of applicants, reviewers, and agency staff.
• Effective. The selection process must achieve the objectives set for it. In practice, these can be summarized as selecting the applications that show the most promise for:
  • Commercializing results, and
  • Helping to achieve the agency’s missions,¹⁶
  • While expanding new knowledge, and
  • Providing support for woman- and minority-owned businesses.

The effectiveness of the agencies’ SBIR programs is discussed elsewhere, in the Chapter 4: SBIR Program Outputs. The remaining components of the selection process are discussed in more detail below, along with some other selected topics, including the degree of centralization, and a more detailed review of the NIH approach based on outside reviewers.

5.4.2. Approaches to Award Selection

Key questions in discussing award selection concern the use of outside reviewers, and who makes final decisions on funding.

5.4.2.1. Use of External Reviewers

DoD. DoD components do not use external reviewers.¹⁷ Two or three technical experts at the laboratory level review each proposal. Proposals are judged competitively on the basis of scientific, technical, and commercial merit, in accordance with the criteria listed in Box 5-2. Prior to the closing of the solicitation, the responsibility for each topic has been clearly established, so reviewers can access “their” proposals immediately after the closing. If a proposal is beyond the expertise of the designated reviewers, the person with overall topic responsibility will obtain additional reviewers.

BOX 5-2

DoD Evaluation Criteria. Evaluation Criteria—Phase I The DoD Components plan to select for award those proposals offering the best value to the government and the nation considering the following factors.

NIH. NIH relies exclusively on external reviewers to provide a technical and commercial assessment of proposals. A few applications received for specific technologies being sought by the agency (via the RFA mechanism) are evaluated at the proposing unit, but even then, outside reviewers provide much of the technical input.

NASA. NASA does not use outside reviewers. Initial technical reviews are done by a panel of technical experts convened at the relevant NASA Centers. Program staff at headquarters subsequently review the awards for alignment with NASA objectives and for other purposes.

DoE. DoE uses internal experts recruited for each specific proposal from across the agency to provide technical and commercial evaluations. After receiving the completed reviews, proposals are scored by the Technical Topic Manager (TTM). Only proposals scoring at least +2 (out of a possible score of +3) are eligible for selection. The SBIR office reviews for discrepancies between the average reviewer score and the TM’s score, and resolves any conflict.

Portfolio Managers for technical program areas select the awardees from that program area. Different programs within DoE have differing philosophies and mechanisms for award selection. Some provide equal awards to each of the program area’s topics. Others select applications with the highest score. DoE tries to decentralize decision making, so that those with the greatest knowledge of the technology make the decisions.

NSF. NSF uses at least three outside (non-NSF) reviewers for each application. Their ratings are then used to guide the Program Officer in making funding decisions. Final approval follows review by the Division of Grants and Agreements.

5.4.2.1.1. Outsider-dominated Processes: The NIH Approach

The peer review process at NIH is by far the most elaborate of all the agencies. It is operated primarily through the Center for Scientific Research (CSR), a separate IC at NIH which serves only the other ICs—it has no direct funding responsibilities of its own.

Study Sections. Applications are received at CSR and are assigned to a particular study section¹⁸ based on the technology and science involved in the proposed research. Panels can either be permanent panels chartered by Congress, or temporary panels designated for operation by NIH. Most SBIR applications are assigned to temporary panels, many of which specialize in SBIR applications only. This trend appears to be accelerating, as the requirements for assessing SBIR applications—notably the commercialization component—are quite different from the basic research conducted under most NIH grants.

Special emphasis panels (SEPs) (temporary panels) are reconstituted for each funding cycle. Almost all SBIR applications are addressed by SEPs, which have a broader technology focus, and less permanent membership, than chartered panels. Many SEPs tend to draw most of their applications from a subset of ICs. For example, the immunology Integrated Review Group (IRG) covers about 15 ICs, but 50 percent of its work comes from the National Institute of Allergy and Infectious Diseases (NIAID), with a further 33 percent from the National Cancer Institute (NCI).

Procedures. Applications are assigned to a subset of the panel—two lead reviewers and one discussant. These panelists begin by separating out the bottom half of all applications. These receive a written review, but are not formally scored. The top 50 percent are scored by the three reviewers and then, after discussion, by the whole panel.

Panel Makeup. NIH guidelines are that at least one panelist should have small business background. One current panel, for example, had 13 small business representatives out of 25 panelists. This is a recent change; previous panels in this technical area had been dominated by academics. NIH guidelines mandate that panels have 35 percent female and 25 percent minority participation.

Scoring is based on five core criteria:

• Significance
• Approach
• Innovation
• Investigators
• Environment

No set point values are assigned to each. Scores are averaged (no effort is made to smooth results for example by eliminating highest and lowest scores), and range between 100 (best) and 500 (worst). Winning scores are usually in the 180-230 range or better, although this varies widely by IC and by funding year. Scores are computed and become final immediately.

Reviewers and Funding Issues. Reviewers are specifically not focused on the size of the funding requested; they are tasked to review whether the amount is appropriate for the research being proposed. As a result, the question of trade-offs between a single larger award against multiple smaller awards is not addressed. On the other hand, this approach permits applicants to propose projects of larger scope and value to the agency mission.

Reviewers and Outcomes of Prior Awards. If the proposing company has received more than 15 Phase II awards, it must note that on its applications. Otherwise, the application forms have no place to list previous awards. Companies with strong track records try to make sure that this is reflected in the text of their application. However, there is no formal mechanism for indicating the existence of, or outcomes from, past awards. Reviewers also do not know the minority or gender status of the PI or of the company’s owners.

Positive and Negative Elements of Outside Review. On the positive side, strong outside review generates a range of benefits. These include:

• The strong positive endorsement for applications that comes from formal peer review.
• The alignment of the program with other peer-reviewed programs at NIH.
• The perception of fairness related to outside review in general.
• The absence of claims that awards are written specifically or “wired” for particular companies.
• The probability that reviewers will have technical knowledge of the proposed award area.

On the negative side, recent efforts to infuse commercialization assessment into the process have had mixed results at best. Problems include:

• Quality of reviews decreasing as workload increases.
• Perceptions of random scoring.
• Conflict of interest problems related to commercialization.¹⁹
• Substantial delays in processing.
• Growing questions about the trade-offs between different size awards.

Overall, outside review adds fairness but also adds complexity and possibly delay. The generic NIH selection process has not yet been substantially adjusted to address the needs of companies trying to make rapid progress in an increasingly competitive environment. Delays that might be appropriate at an institution focused on academic and basic research, may be less applicable to smaller businesses working on a much shorter development cycle.

5.4.3. Fairness

The fairness of selection procedures depends on several factors. These include:

• Transparency—is the process well known and understood?
• Implementation—are procedures always followed?
• Checks and balances—are outcomes effectively reviewed by staff with the knowledge and the authority to correct mistakes?
• Conflict of interest—are there procedures in place and effectively implemented to ensure that conflicts of interest are recognized and eliminated?
• Appeals and resubmissions—are there effective appeals and/or resubmission procedures in place?
• Debriefings—do agency staff give both successful and unsuccessful applicants good feedback about their proposals?

Each agency has strengths and weaknesses in these areas. They are summarized below.

5.4.4. Efficiency

For programs to work well, they must be efficient in term of their use of resources, both internal and external to the agency.

The efficiency objective must be balanced against other considerations. Hypothetically, having one awarding officer making instant decisions would be highly efficient, but would be neither fair nor necessarily effective.

Still, it is possible to construct indicators or measures of efficiency from the perspective of both the applicant and the agency. While many of the efficiency considerations listed in Box 5-3 are a factor in agency program management, it is also noteworthy that no agency has conducted an analysis of program efficiency from the perspective of applicants and award winners. Also, no agency has, to our knowledge, conducted any detailed surveys of customer satisfaction among these populations, focused on program improvement.

BOX 5-3

Possible Efficiency Indicators for the SBIR Selection Process. External: Efficiency for the Applicant Time from application to award

5.4.5. Other Issues

5.4.5.2. Degree of Centralization

It is somewhat misleading to talk about the degree of centralization. Programs handle selection so differently that the same functions are not performed at all agencies. However, it is possible to identify some shared functions (see Table 5-1).

TABLE 5-1

Comparing Selection Procedures Across Agencies.

5.4.6. Selection—Conclusions

We return to our initial conceptual framework of assessment. Are the selection procedures operated at the various agencies meeting the core requirements in this area? Are they:

• Fair.
• Open.
• Efficient.
• Effective.

Fairness. From discussions with applicants and staff, it would appear that fairness is substantially enhanced by two key characteristics:

• Transparency. The more applicants understand about the process, the fairer it appears. Generally, the debriefing programs provide sufficient feedback to meet this requirement, and the agency’s procedures are reasonably well-known. NIH is unique in listing the names of outside panel members.
• Conflict of Interest. Conversely, the more outsiders are used for assessment, the more conflict of interest appears to be a potential problem. Claims of problems at NIH cropped up regularly in interviews, and it appears that procedures to address conflicts of interest could be strengthened there.

Openness. There have been comments in interviews that the DoD program favors companies that are known to the program. However, the data (see Chapter 2) indicate that at least one-third of awards go to new winners every year, and interviewees stated that the program was sufficiently open to new ideas and companies. These comments did not appear in relation to other agencies’ SBIR programs

Efficiency. At most agencies, SBIR managers run the program with limited administrative funds. This limitation means lean program operations, but leanness does not necessarily mean that the needs of applicants and the agency are met as effectively as possible. In general, discussions with staff and awardees lead us to conclude that agency budgetary and efficiency concerns tend to trump the needs of applicants. For example, limiting the number of applications from a firm, or tightening topic definitions to reduce the number of applications, are both “efficient” in the sense that they allow agencies to process applications using the minimum amount of administrative funding. Yet both strategies run the risk of eliminating firms’ and technologies’ potential value.

Effectiveness. The effectiveness of the SBIR program is discussed in more detail in Chapter 4: SBIR Program Outputs. However, effectiveness can also be judged partly by the number of applications the program receives. Application numbers continue to rise.

5.5.1. The Standard Model

Agencies using the standard model differ in the timing of milestones and the handling of administrative material. DoE, for example, uses a 9-month Phase I; NASA staff note that use of a sophisticated electronic tool for submission and program management has worked to reduce the gap between Phase I and Phase II.

Yet despite these differences, which can have an important impact on both awardees and program performance, the basic structure of the system can be captured (see Figure 5-2).

FIGURE 5-2

Standard model for SBIR award cycles.

Key features of this model include:

• A single annual submission date (usually).
• No gap-funding mechanisms between Phase I and Phase II.
• No Fast Track program.
• No Phase IIB funding.

5.5.2. The Gap-reduction Model

The gap-reduction model looks quite different. It includes some or all of a range of features designed to reduce gaps and improve the time performance from initial conception to final commercial product.

Drawing on the efforts made in this direction at the NIH and some components of DoD, as well in some respects at NSF, we can extract the following elements of a gap-reduction approach to award cycles.

5.5.3. Conclusions

While the standard selection model has the virtues of simplicity and of close adherence to SBA guidelines, there are solid reasons for supporting efforts to close the funding gaps that recipient companies confront, especially where this can be done through program redesign with relatively minimal additional costs.

Best-practice analysis across agencies should enable significant improvements in this area if agencies commit to the gap-reduction model, though such a commitment may require the provision of additional administrative funding.

5.6.1. Size/Duration of Phase I

As Table 5-4 shows, the formal size and duration of Phase Is vary considerably at the five agencies and, in the case of DoD, between its components.

TABLE 5-4

Phase I Awards.

Most of the deviations from the $100,000/6-month norm are accounted for by DoD components that reserve $30,000 of the award for an option to cover the gap between the end of Phase I funding and the start of Phase II.

5.6.1.1. Larger Awards at NIH

NIH uses Phase I awards differently than the other agencies:

• It has begun to make much larger awards in some cases.
• It regularly extends Phase I awards to one year (at no additional cost), and has begun to offer a second year of Phase I support in some cases, compared to strict six or nine month limits at most other agencies.
• It provides administrative supplements that boost Phase I awards, when additional resources are needed to complete the proposed research.

Figure 5-3 shows that starting in 1999, the average size of NIH Phase I awards reached $171,000 in 2005. In a growing number of cases, NIH provided Phase I funding of more than $1 million, although the median size of award has grown much more slowly than the average.

FIGURE 5-3

Average size of Phase I awards at NIH, FY1992-2005. SOURCE: NIH Awards Database.

No specific policy decision appears to have been taken to make these larger awards, and the large Phase I awards have been explained in various ways by different staff at NIH. (See NIH Report: Chapter 2³², for detailed discussion).

The median award size has remained close to the guideline, although it continues to trend up, while the mean award size has increased quite rapidly. This implies that extra large awards are (relatively) rare and (relatively) large.

5.6.1.2. Second Year Awards at NIH

Just as the size of awards has changed, NIH has extended the period of support as well. In FY2002 and FY2003, more than 5 percent of all Phase I awards were receiving a second year of support, with a median value of about $200,000 for the second year alone (see Figure 5-4).

FIGURE 5-4

Phase I, year two awards at NIH. SOURCE: NIH Awards Database.

NIH staff and recipients noted in interviews that six months is often too short to complete many biomedical Phase I projects, and that at NIH it is standard practice to grant a “no-cost” extension to one year or even more. This extends the term of the award without providing additional funding. No other agency offers such a liberal extension program.

5.6.2. Size/Duration of Phase II

While most agencies follow the $750,000 limit mandated by SBA, others such as NSF offer less, while NIH offers more. In each case, this is an agency decision, although a waiver from SBA is required for funding beyond $750,000. NIH has received a “blanket” SBA waiver for all extra sized awards.

Beyond the formal limits, NIH once again is an important exception, as it has begun to offer some awards that are much bigger than the norm, and it also extends support far beyond 24 months.³³ GAO’s 2006 report indicates that using a slightly different methodology than NRC analysis, it found that more than half of FY2001-2004 NIH awards were above the guidelines, as were about 12 percent of DoD awards (see Table 5-6).³⁴

TABLE 5-6

Awards Beyond Guidelines at NIH and DoD, FY2001-2004.

Table 5-6 indicates that oversized awards altogether accounted for 69.3 percent of all SBIR funding at NIH (2001-2004) and 22.6 percent of funding at DoD. The larger NIH awards may reflect the rapidly rising cost of medical research (exceeding the inflation rate) and the lack of any change in the guidance for award size by the SBA. The larger awards noted above, reflecting the needs of those agencies, are mirrored by the smaller awards made, for example, by NSF and NASA. In both cases, the award amounts reflect agency judgments on how to best adapt the program to meet their mission needs.

5.6.3. Extra-large Phase II Awards at NIH

The data recording mechanisms at NIH make it extremely difficult to compute overall award sizes with any degree of certainty, as awards are recorded on an annual basis and the tools for linking the different years of a given award are not always accurate.

FIGURE 5-5

Phase II, year one award size at NIH.

SOURCE: National Institutes of Health.

The data show that by 2000, more than two-thirds of Phase II awards at NIH received more than $350,000, and 20 percent received more than $500,000 during the first year of the award. While the average size of the first year of an NIH Phase II award has increased to approximately $500,000, this increase has been driven by some large awards—such as the $3.5 million award in FY2003. Median award sizes have changed less.

Beyond larger awards, NIH also offers longer awards, as Phase II is sometimes continued beyond 2 years (see Figure 5-6).

FIGURE 5-6

Third year of support for Phase II awards at NIH. SOURCE: NIH Awards Database.

The steadily rising numbers of third-year support awards in recent years suggest that they are becoming an important component of NIH SBIR activity. In FY2002 and FY2003, more than 10 percent of awards received a third year of support.

In a few cases, NIH goes further. Ten awards have received a 5th overall year of SBIR support, and a few have received support beyond the fifth year.

5.6.4. Supplementary Funding

NIH is also a leader in finding flexibility within the guidelines for supplementary funding. No other agency provides supplementary funding to cover expected costs.

At NIH, there are two kinds of supplementary funding:

• Administrative supplements.
• Competing supplements.

Each IC has its own guideline for supplements. Some will not permit them at all, and others allow them for only limited amounts. Administrative supplements of up to $50,000 can be authorized by an IC without going to the IC’s governing council.

Only a few ICs allow competing supplements, mainly because these must be re-reviewed by peer reviewers. They reflect large adjustments for change in scope from the original project.

All supplemental requests are considered formal and require documentation. A full application is required for competing supplements, and a budget page and letter justification is required for administrative supplements. However, discus sions with awardees who have received administrative supplements suggest that agency staff have considerable discretion.

The data indicates that the trend for both the number and size of supplementary awards are growing at NIH (see Figure 5-7).

FIGURE 5-7

Supplementary Phase II, year one awards at NIH. SOURCE: NIH Awards Database.

5.6.5. Bridge Funding to Phase III

Traversing the “Valley of Death” between the end of Phase II research funding and the commercial marketplace is the single most important challenge confronting SBIR companies, as they seek to convert research into viable projects and services. As a result, success in helping awardees traverse this crucial period should be a major consideration in assessing each agency’s program.

All the agencies are now using mechanisms for addressing the Phase III/commercialization problem, such as the training and support services described in the section below on commercialization.

In addition, though, some agencies are developing funding mechanism to bridge transition to the market. Two models can be identified:

• The NSF matching funds model (also used in parts of DoD, where it known as Phase II+).
• The NIH focus on support for addressing regulatory requirements.

5.6.5.2. The NIH Regulatory Support Model

NIH has noticed that some of its SBIR awardees have difficulty in dealing with the financial requirements for addressing the FDA regulatory approval process for medical products and devices. The process of regulatory approval is time-consuming and, in some cases, extremely expensive. Deep-pocket investors from the private sector often do not wish to invest in high-risk products before at least some of the regulatory hurdles have been overcome.

Figure 5-8 shows that of the 768 respondents to the 2003 NIH Phase II recipient survey, about 40 percent required FDA approval, but only about 15 per cent had received it. It should be understood that clinical trials involve a whole series of activities, testing both the efficacy and the safety of proposed drugs and devices, using first animal trials in many cases and then human subjects. This is generally not a short or inexpensive process.

FIGURE 5-8

FDA requirements and NIH survey respondents. SOURCE: National Institutes of Health, National Survey to Evaluate the NIH SBIR Program: Final Report, July 2003.

BOX 5-4Characteristics of the Army Phase II Plus Awards

• 126 total Phase II plus awards
• 8 awards greater than the stated maximum
• 51 awards at $250,000 or more
• Median award size: $160,000
• Average award size: $165,670
• Average match: $198,416
• Private investors—35
• Other government agencies—6

What remains is a substantial funding gap, between the end of Phase II— which at NIH may not even create a product that is ready for first phase clinical trials—and the possible adoption of the product by subsequent investors.

NIH has addressed this gap, in part, through a new program called Competing Continuation Awards (CCAs). Competing Continuation Awards offer up to $1 million annually for up to 3 years to companies that complete Phase II, but require additional support as they enter the regulatory approval process.

The introduction of these awards is still too recent to develop a clear picture of their eventual success or utilization level at NIH. However, they have certainly been greeted with enthusiasm by the awardee community, although recent data from NIH show that only three were awarded in FY2004.³⁶

5.8.1. Commercialization Planning

All agencies now require applicants to submit a commercialization plan with their Phase II application. The extent to which this plan actually affects selection decisions varies widely. At DoE, it accounts for one-sixth of the final score.³⁷ At NIH, the effect varies depending on the make-up and interests of the study section. Several program managers at NIH institutes and centers noted that their study sections paid little heed to commercialization in the selection process. However, they also observed that having to develop a commercialization plan was a valuable exercise for the companies, even if it had no impact on selection.

5.8.2. Business Training

As agency interest in Phase III outcomes has increased, the agencies have begun to see that the technically sophisticated winners of Phase II awards are in many cases inexperienced business people with only a limited understanding of how to transition their work into the marketplace. Most agencies have developed initiatives to address this area, although these efforts are constrained by limited funds. While the transfer of best practices does occur between agencies, the extent to which agencies invent their own solutions to what are, in general, common problems is striking.

5.9.1. Differentiation and Flexibility

It is difficult to overstress the importance of diversity within the SBIR program. In fact, given the differences, it is a mistake to talk about “the” SBIR program at all: There are agency SBIR programs, and they are—and should be—different enough that they must be considered separately, as the NRC has done through the companion agency volumes.

These differences—outlined above and emerging through a comparison of the agency volumes—stem of course partly from the differing agency missions and histories, and partly from the differing cultures within them. It is not our mandate to disentangle the sources of these complexities. The point here is to assert that the diversity of the program—which reflects the flexibility permitted within the SBIR structure, one largely enhanced by SBA interpretations of its guidelines—is fundamental to the program’s success. The fact is that DoD programs are managed to generate technologies for use within DoD. It would make little sense to manage them like NIH programs, which aim to provide technologies that will eventually find their way largely to private sector providers in health care. The different objectives impose different needs and constraints and hence different strategies.

The inescapable conclusion from this is that efforts to standardize the operation of different aspects of the program are likely to prove counterproductive.

A further corollary is that managers need substantial flexibility to adapt the program to the context that they confront within their own agency and to their own technical domains. Nothing in the following points should therefore be taken as asserting the need for a single one-size-fits-all approach. We believe that every agency will find value in considering the suggestions and recommendations in this report—but also that each agency must judge those recommendations in light of its own needs, objectives, and programs.

5.9.2. Fairness, Openness, and Selection Procedures

The evidence suggests that the programs are by and large operated in a fair and open manner. Opportunities for funding are widely publicized, and the process for applying and for selection is largely transparent at all agencies.

It is also true that at some agencies—in particular NIH, where peer reviewers from outside the agency have a predominant role in selection—concerns about possible conflict of interest were widely reported by interviewees. Conflicts of interest may occur in some areas, but the different selection procedures used by the agencies require that this issue be addressed at the agency level, as it is recommended, for example, in the NIH volume. It is by no means a general problem.

While open, the agency selection procedures do differ, yet none of the agencies has made any effort to assess the success of their particular selection methodologies, or to determine ways of piloting improvements in them. Linking outcomes to selection has been done only at DoD, through the use of the Commercialization Index, but efforts to integrate this into selection procedures have not been systematically analyzed.

5.9.3. Topics

In agencies where topics define the proposal boundaries, the topic development process is also an important part of outsider perceptions of the program, and here there is less transparency: in none of the agencies is the process of topic development spelled out clearly on the Web site, which at least at DoD results in perceptions that some topics are designed—wired—for specific companies. As noted above, there is no available evidence to support this claim, not least because some companies have developed expertise, giving them a competitive advantage. Increased transparency in this area might enhance perceptions of the program’s fairness more generally.

At DoD in particular, the agency began making efforts to reduce cycle times and to increase the relevancy of topics, starting in 1997 if not before. These efforts have had an important impact on relevancy, where a growing number of topics—now more than 50 percent DoD-wide—are sponsored by acquisition of fices. At NASA, DoE, and NSF, topics are all drawn from technical experts with area responsibilities in the agencies.

However, two other topic issues have emerged:

• Cycle Time. The DoD process does reduce duplication and enhance relevance, but at the cost of timeliness. Recent initiates to develop quick response topics are therefore a promising way of balancing the needs for effectiveness and speed.
• Breadth. At least two factors seem to drive agencies toward narrow definitions of topics, including definitions that mandate the technical solution to be used as well as the problem to be solved. At DoD, the effort to gain interest from acquisition and other non-research communities could lead to the use of SBIR as a form of off-budget contract research. At NSF and DoE, topics have been narrowed in an effort to reduce the flow of applications to what the agency staff believes to be manageable proportions. Neither of these motives seems justified, and we would suggest that agencies seek to keep their topics as broadly defined as possible, at least in terms of the technical solutions that might be acceptable.

5.9.4. Cycle Time

In interviews with awardees and other observers of the program, concerns about cycle time surfaced regularly, at every agency. The fact is that it takes time to develop topics, publish a solicitation, assess applications, make awards, and in the end sign a contract. And that is just for Phase I.

Despite efforts at some agencies—including DoD, and in other ways DoE—the cycle time issue has not been fully addressed. Agencies have not in general fully committed to a “gap reduction” model where every effort is made to squeeze days out of the award cycle. For small businesses, especially those with few awards and few other resources, cycle time is a huge disincentive to applying for SBIR, and constitutes a significant problem for companies in the program.

Every aspect of cycle time—from lags between Phases to the number of annual solicitations, the breadth of topics and the permeability of topic borders, to contracting procedures—should be monitored, assessed, and, where possible, improved. Currently, agencies are in general insufficiently focused on this issue, as detailed in the agency volumes.

The additional funding recommended below should, in part, be used to address this problem.

5.9.5. Award Size

The fact that award size has not been officially increased since the 1992 reauthorization, and has therefore not kept pace with inflation, in itself raises the question of whether the size of awards should be increased. But this is not a simple question. Related questions include: by what amount? One-time only or possibly tied to inflation? Both Phases? At all agencies?

The primary justification for raising the nominal limits is that the cost of research has increased with inflation, and hence that these limits do not buy the agencies the same amount of research results that the Congress intended when this guidance on award size was first introduced.

Agency staff have offered a range of additional justifications for larger awards. At NIH, which has been most active in experimenting with larger awards, justifications include the need to focus on the highest quality research, the likelihood that more funding will lead to more commercial success, the impact of inflation, the need to support companies through regulatory hurdles, and the possibility that higher funding levels will expand the applicant pool by attracting, in particular, high-quality applicants who currently believe SBIR is too small to justify the effort to apply. Finally, and not insignificantly, there is a relatively higher overhead cost of administering more, smaller awards.

The latter is tied to the minimal administrative funding for SBIR, discussed below. The other NIH-specific points are discussed in the NIH volume.

Awardees in interviews have also favored larger awards—until they are asked to make an explicit trade-off between the size of awards and the number of awards. At that point, awardees often become less supportive of larger awards.

There are also arguments against larger (and fewer) awards. Because it is extremely difficult to predict which awards will generate large returns (commercial or otherwise), it may be wise to spread the awards as widely as possible. SBIR awards also play a critical role (described in Chapter 4: SBIR Program Outputs) in supporting the transition of research from the academy to the market place. This kind of motivation may not require more than the existing level of support. There is as yet also no evidence to support the assertion that larger awards generate larger returns, although further analysis at NIH might test that connection.

If we conclude—as we do—that the SBIR programs at the agencies do work as intended by Congress, and do generate significant benefits, we should recommend change only with caution. It therefore seems that while there is a case to increase award size, there are risks involved, and it would be prudent for agencies taking this step to increase the awards incrementally over, perhaps, three years to avoid a sharp contraction of the program and to allow hope for increases in R&D funding to mitigate the impact on applicant success rates of increasing award sizes.

5.9.6. Multiple-Award Winners

Multiple-award winners do not appear to constitute a problem for the SBIR program at any agency. At all agencies except DoD, only a limited number of companies win a sufficiently large number of awards to meet even the loosest definition of a “mill.”

Even at DoD, we find arguments aimed at limiting a company’s participation in SBIR to be unconvincing, for a number of reasons:

1. Successful Commercialization. Aggregate data from the DoD commercialization database indicates that the basic charge against “mills,” i.e., no commercialization, is simply incorrect. Companies winning the most awards are on average more successful commercializers than those winning fewer awards.
   • While data from this source are not comprehensive, they do cover the vast majority of MAWs—and the data indicate that on average, firms with the largest number of awards commercialize as much or more than all other groups of awardees; that in the aggregate, there is no MAW problem of companies living off SBIR awards.
2. For some multiple winners, at least, even though they continue to win a considerable number of awards, the contribution of SBIR to overall revenues has declined.⁴⁶
3. Case studies show that some of the most prolific award winners have successfully commercialized, and have also in other ways met the needs of sponsoring agencies.
4. Graduation. Some of the biggest Phase II winners have graduated from the program either by growing beyond the 500-employee limit or by being acquired—in the case of Foster-Miller, for example, by a foreign-owned firm. Legislating to solve a problem with companies that are in any event no longer eligible seems inappropriate.
5. Contract Research. This can be valuable in and of itself. Agency staff indicate that SBIR fills multiple needs, many of which do not show up in sales data. For example, efficient probes of the technological frontier, conducted on time, on budget, to effectively test technical hypotheses, may save extensive time and resources later, according to agency staff.
6. Spin-offs Some MAWs spin off companies—like Optical Sciences, Creare, and Luna. Creating new firms can be a valuable contribution.
7. Valuable Outputs. Some MAWs have provided the highly efficient and flexible capabilities needed to solve pressing problems rapidly.
8. Compared to What? Agency programs do not impose limits. It is hard to see why small businesses should be subjected to limits on the number of awards annually when successful universities and prime contractors are not subject to such limits.

All these points suggest that while there have been companies that depend on SBIR as their primary source of revenue for a considerable period to time, and there are some who fail to develop commercial results, the evidence strongly supports the conclusion that there is no multiple winner problem. Moreover, those who advocate a limit on the annual number of awards to a given company should explain how this limit is to be addressed across multiple agencies, and why technologies that may be important and unique to a given company should be excluded on this basis.

Given that SBIR awards meet multiple agency needs and multiple congressional objectives, it is difficult to see how the program might be enhanced by the imposition of an arbitrary limit on the number of applications per year, as is currently the case at NSF. However, if agencies continue to see issues in this area, they should consider adopting some version of the DoD “enhanced surveillance” model, in which multiple winners are subject to enhanced scrutiny in the context of the award process.

5.9.7. Information Flows

The shift toward Web-based information delivery has occurred unevenly at the different agencies. DoD has perhaps moved farthest; along with NASA, it was the first agency to require electronic submission of applications, and the online support for applicants is strong. It is, moreover, well integrated with non-electronic information sources, with two innovations in particular being well-received by awardees:

• The Pre-release Period, during which topics are released on the Web along with contact information for the topic authors. This enables potential applicants to directly determine how well their proposed research will fit with the agency’s needs, and provides opportunities for tuning applications, so they are a better fit. This innovation also connects applicants specifically to the technical officers running a particular topic, who will, in the end, also make finding decisions.
• The Help Desk, which is staffed by contractors and is designed to divert non-technical questions (for example about contracts and contracting) to staff with relevant experience in the SBIR program, who may well know these materials better than topic authors for example. Program managers at NIH and NASA have complained in interviews that SBIR applicants require much more help than academic applicants for other kinds of awards. Some of that burden might be alleviated with a better resourced help desk function.⁴⁷

Overall, the growing size of the program, the clear interest exhibited by state economic development staff, and the increasingly positive view of SBIR at many universities, suggest that knowledge about the program is increasingly being diffused to potential applicants. The rise of the Internet—and the high quality Web sites developed by the agencies—mean that general interest can be translated into specifics quickly and inexpensively for both applicants and agencies. This conclusion is buttressed by the continuing flow of new companies into the program; these new companies account for more than 30 percent of all Phase I awards at every agency every year.

Thus it appears that the general outreach function historically fulfilled by the SBIR Agency Coordinators/Program Managers may now be changing toward a more nuanced and targeted role, focused on enhancing opportunities for underserved groups and underserved states, or on specific aspects of the SBIR program (e.g., the July 2005 DoD Phase III meeting in San Diego)—while relying on the Web and other mechanisms to meet the general demand for information. This seems entirely appropriate.

5.9.8. Commercialization Support

While some agencies have been working to support the commercialization activities of their companies for a number of years, this has clearly become a higher priority at most agencies in the recent past. Congress has always permitted agencies to spend a small amount per award ($4,000) on commercialization support, and most agencies have done so.

Commercialization support appears likely to have a significant pay-off for the agencies, partly because many SBIR firms have limited commercial experience. They are often founded by scientists and engineers who are focused on the technology, and interviews with awardees, agency staff, and commercialization contractors all indicate that the business side of commercial activities is often where companies experience the most difficulty.

Important recent initiatives include the extensive set of services provided at Navy through the TAP, and the NIH commitment to roll out the CAP program. These add to the long-running program at DoE.

It is important to understand that the character of commercialization differs quite fundamentally between DoD, NASA, and the remaining nonprocurement agencies respectively (DoE is partly a procurement agency, but for our purposes here, it purchases such a small amount of SBIR outputs for internal consumption, that it is best grouped with the nonprocurement agencies).

At DoD, where the agency provides a substantial market if companies can find a connection to the acquisitions programs, the critical focus of commercialization is on bridging the gaps to adequate Technology Readiness Levels and on finding ways to align companies with potential downstream acquisition programs.

At NASA, where the market within NASA for technologies, though important, may not be large enough to sustain long term development and profitability, the focus is increasingly on the spinout of technologies into the private sector.

At NIH, NSF, and DoE, commercialization means finding markets in the private sector.

These differences mean that while in general all commercialization assistance programs provide help in formulating business plans, in developing strategic business objectives, and in tuning pitches for more funding, there are important differences. In particular, DoD, which accounts for about half of the entire SBIR program, has commercialization programs that are largely (though not exclusively) focused on markets internal to DoD and on the particularly complex process of finding a way into the acquisition stream. This requires different training, different analysis, and different benchmarks than do other commercialization programs.

All of these suggest that it is important to find appropriate benchmarks against which to measure success.