Starting Date and Coverage

The survey of SBIR Phase II awards was administered in 2005, and included awards made through 2001. This allowed most of the Phase II awarded projects (nominally two years) to be completed, and provided some time for commercialization. The selection of the end date of 2001 was consistent with a GAO study, which in 1991, surveyed awards made through 1987.

A start date of 1992 was selected. The year 1992 for the earliest Phase II project was considered a realistic starting date for the coverage, allowing inclusion of the same (1992) projects as the DoD 1996 survey, and of the 1992, and 1993 projects surveyed in 1998 for SBA. This adds to the longitudinal capacities of the study. The 10 years of Phase II coverage spanned the period of increased funding set-asides and the impact of the 1992 reauthorization. This time frame allowed for extended periods of commercialization and for a robust spectrum of economic conditions. Establishing 1992 as the cut-off date for starting the survey helped to avoid the problem that older awards suffer from several problems, including meager early data collection as well as potentially irredeemable data loss; the fact that some firms and PIs are no longer in place; and fading memories.

Award Numbers

While adding the annual awards numbers of the five agencies would seem to define the larger sample, the process was more complicated. Agency reports usually involve some estimating and anticipation of successful negotiation of selected proposals. Agencies rarely correct reports after the fact. Setting limitations on the number of projects to be surveyed from each firm required knowing how many awards each firm had received from all five agencies. Thus, the first step was to obtain all of the award databases from each agency and combine them into a single database. Defining the database was further complicated by variations in firm identification, location, phone numbers, and points of contact within individual agency databases. Ultimately, we determined that 4,085 firms had been awarded 11,214 Phase II awards (an average of 2.7 Phase II awards per firm) by the five agencies during the 1992–2001 time frame. Using the most recent awards, the firm information was updated to the most current contact information for each firm.

Sampling Approaches and Issues

The Phase II survey used an array of sampling techniques, to ensure adequate coverage of projects to address a wide range both of outcomes and potential explanatory variables, and also to address the problem of skew. That is, a relatively small percentage of funded projects typically account for a large percentage of commercial impact in the field of advanced, high-risk technologies.

• Random samples. After integrating the 11,214 awards into a single database, a random sample of approximately 20 percent was sampled. Then a random sample of 20 percent was ensured for each year; e.g., 20 percent of the 1992 awards, of the 1993 awards, etc. Verifying the total sample one year at a time allowed improved ability to adapt to changes in the program over time, as otherwise the increased number of awards made in recent years might dominate the sample.
• Random sample by agency. Surveyed awards were grouped by agency; additional respondents were randomly selected as required to ensure that at least 20 percent of each agency’s awards were included in the sample.
• Firm surveys. After the random selection, 100 percent of the Phase IIs that went to firms with only one or two awards were polled. These are the hardest firms to find for older awards. Address information is highly perishable, particularly for earlier award years. For firms that had more than two awards, 20 percent were selected, but no less than two.
• Top performers. The problem of skew was dealt with by ensuring that all Phase IIs known to meet a specific commercialization threshold (total of $10 million in the sum of sales plus additional investment) were surveyed (derived from the DoD commercialization database). Since 56 percent of all awards were in the random and firm samples described above, only 95 Phase IIs were added in this fashion.
• Coding. The project database tracks the survey sample, which corresponds with each response. For example, it is possible for a randomly sampled project from a firm that had only two awards to be a top performer. Thus, the response could be analyzed as a random sample for the program, a random sample for the awarding agency, a top performer, and as part of the sample of single or double winners. In addition, the database allows examination of the responses for the array of potential explanatory or demographic variables.
• Total number of surveys. The approach described above generated a sample of 6,410 projects, and 4,085 firm surveys—an average of 1.6 award surveys per firm. Each firm receiving at least one project survey also received a firm survey. Although this approach sampled more than 57 percent of the awards, multiple-award winners, on average, were asked to respond to surveys covering about 20 percent of their projects.

Administration of the Survey

The questionnaire drew extensively from the one used in the 1999 National Research Council assessment of SBIR at the Department of Defense, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative.¹ That questionnaire in turn built upon the questionnaire for the 1991 GAO SBIR study. Twenty-four of the 29 questions on the earlier NRC study were incorporated. The researchers added 24 new questions to attempt to understand both commercial and noncommercial aspects, including knowledge base impacts, of SBIR, and to gain insight into impacts of program management. Potential questions were discussed with each agency, and their input was considered. In determining questions that should be in the survey, the research team also considered which issues and questions were best examined in the case studies and other research methodologies. Many of the resultant 33 Phase II Award survey questions and 15 Firm Survey questions had multiple parts.

The surveys were administered online, using a Web server. The formatting, encoding, and administration of the survey was subcontracted to BRTRC, Inc., of Fairfax, VA.

There are many advantages to online surveys (including cost, speed, and possibly response rates). Response rates become clear fairly quickly, and can rapidly indicate needed follow up for nonrespondents. Hyperlinks provide amplifying information, and built-in quality checks control the internal consistency of the responses. Finally, online surveys allow dynamic branching of question sets, with some respondents answering selected subsets of questions but not others, depending on prior responses.

Prior to the survey, we recognized two significant advantages of a paper survey over an online one. For every firm (and thus every award), the agencies had provided a mailing address. Thus, surveys could be addressed to the firm president or CEO at that address. That senior official could then forward the survey to the correct official within the firm for completion. For an online survey we needed to know the email address of the correct official. Also, each firm needed a password to protect its answers. We had an SBIR Point of Contact (POC) and email address and password for every firm, which had submitted for a DoD SBIR 1999 survey. However, we had only limited email addresses and no passwords for the remainder of the firms. For many, the email addresses that we did have were those of Principal Investigators rather than an official of the firm. The decision to use an online survey meant that the first step of survey distribution was an outreach effort to establish contact with the firms.

Outreach by Mail

This outreach phase began with the establishing a NAS registration Web site which allowed each firm to establish a POC, email address and password. Next, the Study Director, Dr. Charles Wessner, sent a letter to those firms for which email contacts were not available. Ultimately only 150 of the 2,080² firms provided POC/email after receipt of this letter. Six hundred fifty of those letters were returned by the post office as invalid addresses. Each returned letter required thorough research by calling the agency provided phone number for the firm, then using the Central Contractor Registration database, Business.com (powered by Google) and Switchboard.com to try to find correct address information. When an apparent match was found, the firm was called to verify that it was in fact the firm, which had completed the SBIR. Two hundred thirty-seven of the 650 missing firms were so located. Another ten firms were located which had gone out of business and had no POC.

Two months after the first mailing, a second letter from the Study Director went to firms whose first letter had not been returned, but which had not yet registered a POC. This letter also went to 176 firms, which had a POC email, but no password, and to the 237 newly corrected addresses. The large number of letters (277) from this second mailing that were returned by the postal service, indicated that there were more bad addresses in the first mailing than indicated by its returned mail. (If the initial letter was inadvertently delivered, it may have been thrown away.) Of the 277 returned second letters, 58 firms were located using the search methodology described above. These firms were asked on the phone to go to the registration Web site to enter POC/email/password. A total of 93 firms provided POC/email/password on the registration site subsequent to the second mailing. Three additional firms were identified as out of business.

The final mailing, a week before survey, was sent to those firms that had not received either of the first two letters. It announced the study/survey and requested support of the 1,888 CEOs for which we had assumed good POC/email information from the DoD SBIR submission site. That letter asked the recipients to provide new contact information at the DoD submission site if the firm information had changed since their last submission. One hundred seventy-three of these letters were returned. We were able to find new addresses for 53 of these, and ask those firms to update their information. One hundred fifteen firms could not be found and five more were identified as out of business.

The three mailings had demonstrated that at least 1,100 (27 percent) of the mailing addresses were in error, 734 of which firms could not be found, and 18 were reported to be out of business.

Outreach by Email

We began Internet contact by emailing the 1,888 DoD Points of Contact (POCs) to verify their email and give them opportunity to identify a new POC. Four hundred ninety-four of those emails bounced. The next email went to 788 email addresses that we had received from agencies as PI emails. We asked that the PI have the correct company POC identify themselves at the NAS Update registration site. One hundred eighty-eight of these emails bounced. After more detailed search of the list used by NIH to send out their survey, we identified 83 additional PIs and sent them the PI email discussed above. Email to the POCs not on the DoD Submission site resulted in 110 more POC/email/Password being registered on the NAS registration site.

We began the survey at the end of February with an email to 100 POCs as a beta test and followed that with another email to 2,041 POCs (total of 2,141) a week later.

Survey Responses

By August 5, 2005, five months after release of the survey, 1,239 firms had begun and 1,149 firms had completed at least 14 of 15 questions on the firm survey. Project surveys were begun on 1,916 Phase II awards. Of the 4,085 firms that received Phase II SBIR awards from DoD, NIH, NASA, NSF, or DoE from 1992 to 2001, an additional seven firms were identified as out of business (total of 25) and no email addresses could be found for 893. For an additional 500 firms, the best email addresses that were found were also undeliverable. These 1,418 firms could not be contacted, thus had no opportunity to complete the surveys. Of these firms, 585 had mailing addresses known to be bad. The 1,418 firms that could not be contacted were responsible for 1,885 of the individual awards in the sample.

Using the same methodology as the GAO had used in the 1992 report of their 1991 survey of SBIR, undeliverables and out-of-business firms were eliminated prior to determining the response rate. Although 4,085 firms were surveyed, 1,418 firms were eliminated as described. This left 2,667 firms, of which 1,239 responded, representing a 46 percent response rate by firms,³ which could respond. Similarly when the awards, which were won by firms in the undeliverable category, were eliminated (6,408 minus 1,885), this left 4,523 projects, of which 1,916 responded, representing a 42 percent response rate. Table App-B-1 displays by agency the number of Phase II awards in the sample, the number of those awards, which by having good email addresses had the opportunity to respond, and the number that responded.⁴ Percentages displayed are the percentage of awards with good addresses, the percentage of the sample that responded, and the responses as a percentage of awards with the opportunity to respond.

TABLE App-B-1

NRC Phase II Survey Responses by Agency, August 4, 2005.

The NRC Methodology report had assumed a response rate of about 20 percent. Considering the length of the survey and its voluntary nature, the rate achieved was relatively high and reflects both the interest of the participants in the SBIR program and the extensive follow-up efforts. At the same time, the possibility of response biases that could significantly affect the survey results must be recognized. For example, it may be possible that some of the firms that could not be found have been unsuccessful and folded. It may also be possible that unsuccessful firms were less likely to respond to the survey.

Part I. Current status of the Project

1. What is the current status of the project funded by the referenced SBIR award? Select the one best answer. Percentages are based on the 920 respondents who answered this question.

2. Did the reasons for discontinuing this project include any of the following?

PLEASE SELECT YES OR NO FOR EACH REASON AND NOTE THE ONE PRIMARY REASON

300 projects were discontinued. The % below are the percent of the discontinued projects that responded with the indicated response.

The next question to be answered depends on the answer to question 1. If c, go to question 3. If b, skip to question 16.

Part II. Commercialization activities and planning

Questions 3–7 concern actual sales to date resulting from the technology developed during this project. Sales includes all sales of a product, process, or service, to federal or private sector customers resulting from the technology developed during this Phase II project. A sale also includes licensing, the sale of technology or rights, etc.

3. Has your company and/or licensee had any actual sales of products, processes, services or other sales incorporating the technology developed during this project? Select all that apply. This question was not answered for those projects still in Phase II (5%) or for projects, which were discontinued without sales or additional funding (23%). The denominator for the percentages below is all projects that answered the survey. Only 72% of all projects, which answered the survey, could respond to this question.

From the combination of responses 1b, 3a, and 3b, we can conclude that 31% had no sales and expect none, and that 18% had no sales but expect sales.

4. For your company and/or your licensee(s), when did the first sale occur, and what is the approximate amount of total sales resulting from the technology developed during this project? If multiple SBIR awards contributed to the ultimate commercial outcome, report only the share of total sales appropriate to this SBIR project. Enter the requested information for your company in the first column and, if applicable and if known, for your licensee(s) in the second column. Enter approximate dollars. If none, enter 0 (zero).

5. To date, approximately what percent of total sales from the technology developed during this project have gone to the following customers? If none enter 0 (zero). Round percentages. Answers should add to about 100%.⁵ 920 firms responded to this question as to what percent of their sales went to each agency or sector.

6. Is a federal system or acquisition program using the technology from this Phase II?

If yes, please provide the name of the federal system or acquisition program that is using the technology. 12% reported use in a federal system or acquisition program.

7. Did a commercial product result from this Phase II project? 27% reported a commercial product.

8. If you have had no sales to date resulting from the technology developed during this project, what year do you expect the first sales for your company or its licensee? Only firms that had no sales but answered that they expected sales got this question.

9. For your company and/or your licensee, what is the approximate amount of total sales expected between now and the end of 2006 resulting from the technology developed during this project? (If none, enter 0 [zero].) This question was seen by those who already had sales and those w/o sales who reported expecting sales; however, averages are computed for all who took the survey since all could have expected sales.

10. How did you (or do you expect to) commercialize your SBIR award?

11. Which of the following, if any, describes the type and status of marketing activities by your company and/or your licensee for this project? Select one for each marketing activity. This question answered by 620 firms, which completed Phase II and have not discontinued the project, w/o sales or additional funding.

Part III. Other outcomes

12. As a result of the technology developed during this project, which of the following describes your company’s activities with other companies and investors? (Select all that apply.) Percentage of the 620 who answered this question.

13. In your opinion, in the absence of this SBIR award, would your company have undertaken this project? Select one. Percentage of the 618 who answered this question.

14. If you had undertaken this project in the absence of SBIR, this project would have been Questions 14 and 15 were answered only by the 13% who responded that they definitely or probably would have undertaken this project in the absence of SBIR.

15. In the absence of SBIR funding, (Please provide your best estimate of the impact.)

a. The start of this project would have been delayed about an average of 11 months.

62% of the 81 firms expected the project would have been delayed. 49% (40 firms) expected the delay would be at least 12 months. 22% anticipated a delay of at least 24 months.

b. The expected duration/time to completion would have been

c. In achieving similar goals and milestones, the project would be

16. Employee information. Enter number of employees. You may enter fractions of full-time effort ( e.g., 1.2 employees). Please include both part-time and full-time employees, and consultants, in your calculation.

17. The Principal Investigator for this Phase II Award was a (check all that apply)

18. Please give the number of patents, copyrights, trademarks, and/or scientific publications for the technology developed as a result of this project. Enter numbers. If none, enter 0 (zero). Results are for 816 respondents to this question.

Part IV. Other SBIR funding

19. How many SBIR awards did your company receive prior to the Phase I that led to this Phase II?

a. Number of previous Phase I awards. Average of 22. 29% had no prior Phase I and another 42% had 5 or less prior Phase I.

b. Number of previous Phase II awards. Average of 8. 46% had no prior Phase II and another 36% had 5 or less prior Phase II.

20. How many SBIR awards has your company received that are related to the project/technology supported by this Phase II award?

a. Number of related Phase I awards. Average of two awards. 46% had no prior related Phase I and another 48% had 5 or less prior related Phase I.

b. Number of related Phase II awards. Average of one. 56.5% had no prior related Phase II and another 42.1% had 5 or less prior related Phase II.

Part V. Funding and other assistance

21. Prior to this SBIR Phase II award, did your company receive funds for research or development of the technology in this project from any of the following sources? Of 854 respondents.

22. Have you received or invested any additional developmental funding in this project?

23. To date, what has been the total additional developmental funding for the technology developed during this project? Any entries in the Reported column are based on information previously reported by your firm to DoD or NAS. They are provided to assist you in completing the Developmental Funding column. Previously reported information did not include investment by your company or personal investment. Please update this information to include breaking out Private investment and Other investment by subcategory. Enter dollars provided by each of the listed sources. If none, enter 0 (zero).</eei> The dollars shown are determined by dividing the total funding in that category by the 920 respondents who started the survey to determine an average funding. Only 462 of these respondents reported any additional funding.

24. Did this award identify matching funds or other types of cost sharing in the Phase II Proposal?⁶

25. Regarding sources of matching or co-investment funding that were proposed for Phase II, check all that apply. The percentages below are computed for those 161 projects, which reported matching funds.

26. Did you experience a gap between the end of Phase I and the start of Phase II?

The average gap reported by 584 respondents was 5 months. 3% of the respondents reported a gap of two or more years.

27. Project history. Please fill in for all dates that have occurred. This information is meaningless in aggregate. It has to be examined project by project in conjunction with the date of the Phase I end and the date of the Phase II award to calculate the gaps.

28. If you experienced funding gap between Phase I and Phase II for this award, (select all answers that apply)

29. Did you receive assistance in Phase I or Phase II proposal preparation for this award? Of 791 repondents.

Was this assistance useful?

30. In executing this award, was there any involvement by university faculty, graduate students, and/or university developed technologies? Of 837 respondents.

31. This question addresses any relationships between your firm’s efforts on this Phase II project and any University (ies) or College (s). The percentages are computed against the 837 who answered question 30, not just those who answered yes to question 30. Select all that apply.

In remarks enter the name of the University or College that is referred to in any blocks that are checked above. If more than one institution is referred to, briefly indicate the name and role of each.

32. Did commercialization of the results of your SBIR award require FDA approval?

Yes

2%

In what stage of the approval process are you for commercializing this SBIR award?