3.1. How Panels Were Selected

As expected, the members of each oversight group sought out panel members who had broad understanding of their field and extensive connections with the international research community, including members of the academic, industrial, and government sectors.

The attempt to use discrete categories was complicated by overlap. For example, any person is likely to be both a researcher and a user of research. Similarly, some of the "US" academic researchers were born outside the United States or conducted collaborative research with non-US researchers. Still, about half the panel members selected were US academic researchers in the field. The remaining half were US researchers in related fields, "users" of research (as defined in section 2.3), and international (non-US) researchers.

Of the 12 members of the mathematics panel, three were non-US researchers, including a current and a former president of the respective national academies of sciences in their countries. Two were US researchers employed by industry, and one was a US Nobel Prize-winning chemist who uses mathematics in his research. The remaining six members were US academic researchers in mathematics.

Similarly, the 13 members of the materials science and engineering panel included three non-US researchers, two US researchers-administrators in industry—all in materials science and engineering—and one US researcher in a related field. The remainder were US academic researchers in materials science and engineering.

The 14 members of the immunology panel included three non-US researchers, two US researchers-administrators in industry, and one US policy analyst, in addition to US academic researchers.

The oversight groups, charged with nominating panel members and overseeing the benchmarking experiments, were also diverse in membership, although they did not contain foreign members. Each comprised about six individuals. Two members of each oversight group were members of COSEPUP, and the remainder were representatives of related NRC commissions and boards. For example, the oversight group of the materials science and engineering panel included two COSEPUP members and members of the Commission on Physical Sciences, Mathematics, and Applications (CPSMA), the Commission on Engineering and Technical Systems (CETS), CETS's National Materials Advisory Board (NMAB), and CPSMA's Board on Physics and Astronomy (BPA); among them were four industry researchers and three academic researchers, all from the United States.

3.2. How Panels Assessed Their Fields

Each panel used a variety of methods to assess its field; the methods depended on the disciplines within the field. The methods used included

• The "virtual congress".
• Citation analysis.
• Journal-publication analysis.
• Quantitative data analysis (for example, numbers of graduate students, degrees, and employment status).
• Prize analysis.
• International-congress speakers.

Each method is described in more detail below. The assessments were to be current—that is, the status of US research in the field today, not in the past (most analysis relied on information collected within the past 5-10 years). Some information, such as that provided by the virtual congress, were current.

A challenging aspect of the benchmarking exercise was the great size of the US research enterprise. In the three benchmarked fields, US researchers were found to perform a dominant portion of the world's research, as measured by numbers of publications. Because of this size dominance, the panels often drew comparisons of leadership status in relation to regions or to the rest of the world combined rather than to individual countries.

Another challenge for the panels was to determine which subfields to analyze. Panel members used their own judgment and existing documents. For example, the mathematics panel used the meeting sections of the International Congress of Mathematics and the International Congress of Industrial and Applied Mathematics as starting points. The materials science and engineering panel started with subfields identified in a recent report by the National Science and Technology Council that discussed materials science and engineering across the federal government. The immunology panel faced a greater difficulty: the field is an amalgam of subfields in larger disciplines, and no subfield classification systems existed, so the panel chose to develop its own.

A method of defining subfields that reflected the original purpose of the COSEPUP leadership recommendations was to define subfields as areas within which researchers can move and still work within their realm of expertise. That connects directly with the goal of leadership because it enables a country that is strong in a given subfield to respond to and enter a "hot" new area; US researchers were able to do this, for example, in many areas of materials science and engineering.

3.3. How Panels Characterized US Research

Each panel concluded that the United States was at least among the world leaders in its field. However, each panel also identified subfields in which the United States lagged the world leaders. Each panel identified key infrastructure concerns. The mathematics panel found that the United States was the world leader in the field but that there were "storm clouds on the horizon" because of its heavy reliance on foreign mathematicians who had recently immigrated to the United States. If that influx wanes, the US leadership might be in jeopardy, given the decline in the number of American students applying to graduate school in mathematics. (Although the other panels did not explicitly make that point, more than half the graduate students in science and engineering are foreign-born, so the national research enterprise depends heavily on researchers from other countries.)

The materials science and engineering panel found that the United States was at least among the world leaders in all subfields of materials science and engineering and the leader in some subfields, although neither the United States nor any other country was the world leader in the field as a whole. A general area of US weakness for most subfields was in materials synthesis and processing. Of particular concern in this field was the lack of adequate funding to modernize major research facilities in the United States, many of which are much older than those in other countries, and to build the new facilities needed to maintain research leadership.

The immunology panel found that the United States was the world leader in immunology but that, although US dominance was evident in the major subfields, the United States was only among the world leaders in some parts of subfields. One general concern was the increasing cost of maintaining the mouse facilities that provide a key portion of research infrastructure. Another was the increasing difficulty in locating sufficient numbers of patients for clinical trials.

3.4. Factors Influencing US Performance

Each panel identified the key factors influencing US performance in its field and assessed the factors primarily on the basis of its members' judgment. No quantitative measures were identified that were sufficiently comprehensive to account for performance.

The three panels differed somewhat in their lists of factors considered most important, as might be expected among three widely different fields. However, the three lists overlapped in the following ways:

• Human resources and graduate education—mathematics, materials, and immunology.
• Funding—mathematics, materials, and immunology.
• Innovation process and industry—materials and immunology.
• Infrastructure—materials and immunology.

COSEPUP finds that degree of overlap to be significant, partly because other recent National Academies reports⁹ on science and engineering have identified the importance of these same factors.

3.5. Future Relative Position of US Research

With respect to the likely future position of the United States in research relative to other countries, a more diverse set of factors was identified, largely by qualitative techniques:

• Intellectual quality of researchers and ability to attract talented researchers-mathematics, materials, and immunology.
• Ability to strengthen interdisciplinary research-mathematics and materials.
• Maintenance of strong, research-based graduate education-mathematics, immunology, and materials.
• Maintenance of a strong technological infrastructure-materials.
• Cooperation among government, industrial, and academic sectors-materials.
• Increased competition from Europe and other countries-immunology and materials.
• Effect of the shift toward health-maintenance organizations on clinical research-immunology.
• Adequate funding and other resources-mathematics, materials, and immunology.

Footnotes

8

See, for example, The Quality of the UK Science Base, United Kingdom, Office of Science and Technology, Department of Trade and Industry, March 1997.

9

Examples of these reports include Science, Technology, and the Federal Government: National Goals for a New Era, Evaluating Federal Research Programs: Research and the Government Performance and Results Act, and Allocating Federal Funds for Science and Technology.