The Multiple Uses of Performance Measures

Performance measures have many uses, Feller continued. First, they are used to do retrospective assessments of realized, observed, and measured impacts. In this case, basic questions are: How has that program worked? Has it produced the results for which it was funded? How could these research advances contribute to societal objectives?

Second, performance measures can be used to assess the best direction in which to head. Is this where scientific advances will occur? Will these scientific advances lead to the achievement of societal objectives?

Finally, performance measures can benchmark accomplishments against historical or international measures and advocate for particular actions.

In each of these cases, performance measures have little relevance in the abstract, Feller said. They need to be related to the decisions at hand, and their promise and limitations depend on the decision being made. “They are quite necessary and productive for certain types of decisions, problematic for others, and harmful for others.”

The context of performance measures determines much of their promise and limitations, according to Feller. A critical question is who is asking the questions. In a university setting, a promotion and tenure committee might ask about publications and citations while a dean or president might ask which areas of the university to support. In the federal government, a member of Congress might ask whether appropriations for a particular laboratory will produce jobs in his or her district, the director of OSTP might ask questions about recommendations to make to the President, and the director of the Office of Management and Budget (OMB) might ask about U.S. research expenditures relative to all other demands on the budget. Similarly, different federal agencies might ask different questions. NSF might want to know how to use research to advance the frontiers of knowledge, while the EPA might want to use science to support regulatory decisions.

Performance measures have been the focus of longstanding and diverse research traditions, Feller said. Over the course of four decades, he has studied patent data, bibliometrics, and many other measures related to research performance. The economics literature continues to produce more refined measures, better data, and new estimation techniques. Feller cited one study that used 37 performance measures in terms of outputs, outcomes, and impacts. Scorecards that compile measures, both nationally and internationally, also are proliferating. New theories, models, techniques, and datasets are producing an intellectual ferment in the use of performance measures. In addition, the community of practice is strengthening, which will increase the supply and use of research-based, policy-relevant performance measures. “This is a rich and fertile field for exploration, for discovery, and for development,” Feller observed.

The Promise of Performance Measures

In terms of the promise of performance measures, they provide useful baselines for assessing several forms of accountability.

First, such measures provide evidence that an agency, laboratory, or individual is making good use of allocated funds.

Second, well-defined objectives and documentation of results facilitate communication with funders, performers, users, and others. Results become verifiable and quantifiable information on what has been done.

Performance measures focus attention on the ultimate objectives of public policy. Researchers and policymakers sometimes refer to the “black box” of innovation - the complex process of turning knowledge into applications - and much research done in economics and related disciplines tries to explain what goes on inside the black box.

Finally, performance measures can help policymakers avoid “fads” that direct attention in unproductive ways. Data can document that some phenomena do not have a solid evidentiary base and that it is time to move on.

The Limits of Performance Measures

An obvious limit on performance measures is that the returns on research are uncertain, long term, and circuitous. This makes it difficult to put research into a strict accountability regime. Doing so “loses sight of the dynamics of science and technology,” Feller said.

In addition, impacts typically depend on complementary actions by entities other than the federal government. This is particularly the case as fundamental research moves toward technological innovation, implementation, and practice.

A less obvious limitation is that the benefits from failure are often underestimated by performance measures. Risk and uncertainty are inevitable in research, which means that research often generates negative results. Yet such results can redirect research into extremely productive directions, Feller said.

The selection of performance measure can also offer what Feller called a specious precision. Different measurable outcomes such as productivity, employment, competitiveness, and growth are not necessarily compatible with each other. There may also be tradeoffs among measures, so that greater accuracy in one generates greater uncertainty in the other.

The selection of performance measures can distort incentives. Research managers strive to improve performance on the measures selected, which can lead to results that are not necessarily compatible with longer-term objectives.

A final limitation, according to Feller, is that there is limited public evidence to date of the contributions that performance measurement has made to improve decision making.

Three Major Questions

Federal science policy must ask three big questions, Feller observed:

1. How much money should be allocated to federal research?
2. How much money should be spent across missions, agencies, or fields of research?
3. Which performers should conduct research, and what are the allocation criteria used to distribute these funds?

Performance measures do not provide a basis for answering the first of these questions. They do not indicate if the ratio of R and D to gross domestic product (GDP) should be 2.8 percent, 3 percent, 3.2 percent, 4 percent, or 6 percent. “I don’t know if there is any evidence to support one level rather than the other,” said Feller.

With regard to the allocation of money across fields, performance measures lead to multiple answers and therefore to multiple possible decisions. For example, bibliometric studies among journals might point toward the importance of biochemistry, economic research might point to the influence of computer engineering, and survey research on the use of scientific knowledge by industry might point to the need to support engineering and applied research fields. Of course, all scientific fields are connected to others, but that does not help make decisions about where to increase funding at the margin. “Depending on the methodology and the performance measures you use, you get different fields of science that tend to be emphasized,” said Feller.

Performance measures have greater potential, Feller continued, in deciding among the performers of research, whether universities, government laboratories, non-governmental organizations, or other research institutes and among investigators. Agencies often have to make such decisions, along with decisions about the structure of research teams and centers. However, performance measures are currently underused for this purpose.

Do No Harm

It is critically important to “do no harm,” Feller emphasized. A major goal of developing performance measures is to improve the quality of decision making. But there are dangers in relying too heavily on performance measures. For example, some states are discussing the use of performance measures to determine funding levels for higher education, despite their many limitations. Some policymakers “are moving pell-mell into the Slough of Despond, and I think that’s what you want to avoid.”

Policy analysts also must be careful not to overpromise what performance measures can do. Analysts will be called to account if their measures turn out to be mistaken and lead to harmful decisions, Feller concluded.

The Features of Complex Systems

Case studies of the role of research in innovation reveal an extremely complex process in which research is an important element of the process but not the only important element. “Everything is connected to everything else,” said Sarewitz. “It’s an ecosystem, and all things flow in different ways at different times depending on who is looking when and where in the process.” For example, technology often enables basic science to address new questions. Similarly, tacit knowledge acquired through the day-to-day practice of, for example, engineers or physicians can raise important questions for researchers. As an example, Sarewitz cited a statement by former NIH Director Harold Varmus that some cancer treatments are “unreasonably effective” but that it is hard to fund research on these treatments because such research is considered high risk. “I was stunned by this, because my view of the complexity of the innovation system is that if we understand that technologies and practices themselves are sources of problems that research can address, then one ought to see unreasonably effective cancer treatments as an incredibly potent attractor of research.” However, the predominant model of research pursued at NIH is to understand the fundamental dynamics of a disease, which then will lead rationally toward the best treatments to use.

There is a deeper problem, said Sarewitz. In a complex system such as the innovation ecosystem, there is no reason to believe that optimizing the performance of any one part of the system will optimize or even necessarily improve the performance of the system as a whole. “Another way to put this is that research is not an independent variable in the innovation system. We generally don’t know what the independent variables are. For analytical purposes there may not be any.”

The connections that link the elements of the innovation system represent contextual factors that can be crucial determinants of performance. Factors such as trust among the people in an institution, administrative structures that allow for rapid learning and adaptation, or historical ties between different institutions that allow them to work together can be very important for determining the dynamics and ultimate success of complex innovation processes. These sorts of internal systems dynamics can be teased out through careful case studies, Sarewitz said. But they are very difficult to capture in de-contextualized and rigid performance measures.

The Policy Perspective

Policymakers have an array of tools that they can use to try to influence the behavior of complex innovation processes. However, just a few of these tools relate directly to research, and the relations among these tools are poorly understood. For example, analysts would have difficulty measuring and comparing the performance of intramural laboratories and extramural university research without also knowing the institutional contexts of the research performers.

More generally, research performance measures may reveal little about the value and contextual appropriateness of the full array of science policy tools. For example, tools like demonstration and procurement, especially as done by the Department of Defense, have been enormous drivers of innovation in the past, yet they are outside the domain of research performance measures. Given the importance of other factors, optimizing research performance could lead to undesired outcomes.

These undesired outcomes may even have ethical and moral dimensions, said Sarewitz. For example, policy decisions in the early 1980s accelerated the privatization of the results of publicly funded research and helped to elevate the importance of patents as an apparent indicator of innovation. However, these policy decisions have consequences that bear on equity to access of some of the products of publicly funded research. In the medical arena, to cite an example Sarewitz mentioned, they could have slowed innovation in socially important domains of research, such as the development of agricultural biotechnologies for developing countries.

Innovative Approaches

The science and technology policy and research communities have to engage as imaginatively as possible in expanding the array of approaches used to understand, assess, and talk about innovation processes and their outcomes in society, Sarewitz said. First, new understandings of complex innovation processes can be used to help improve policy making. Case studies, for example, can produce synthetic systems-oriented insights that can have a powerful and enriching impact on policy making and “hopefully, change the narrative.”

Second, the science policy research community can do a better job of coming up with diverse performance criteria and measures that can support rather than displace qualitative insights. An interesting recent example involved the public policy analogues of market failures, which could be used to drive public investments in the same way that market failures have in the past (Bozeman and Sarewitz, 2005). “We don’t know yet if this particular approach is going to turn out to be a valuable tool,” said Sarewitz. “The point I’m trying to make is that the narrow array of things we are now measuring as indicators of performance of the innovation system, mostly matters of research productivity, is impoverished and we can and should do better.”

Research is crucially important in innovation, Sarewitz concluded. But its importance is contextual and contingent in space, among institutions, and over time. “If decision makers focus on optimizing performance and the innovation enterprise based on measures that largely deal with research, research performance, and research outputs, they’ll likely fail to achieve the goals that the public expects from the nation’s R and D investment.”

Accounting for Overlooked Measures

Greater transparency can help overcome these challenges, said Spector. The growth of modern communication technologies makes transparency much more feasible today than in the past, providing a more open view of research outcomes. Similarly, better visualizations can produce representations that are useful to policymakers and the public in assessing the value of research.

One of the most important products of research, though it is sometimes overlooked, is the training of people, Spector said. “If you talk to most of my peers in industry, what we really care about as much as anything else is the immense amount of training that goes on through the research that’s done.” For example, venture capitalists would rate talent as the most important input into innovation.

Also, the diversity of research approaches can be an important factor in research. In computer science, for example, funding has come not only from the NSF, in which peer review largely determines what science will be done, but also from the Defense Advanced Research Projects Agency, which has a much more mission-oriented approach. “DARPA has made huge bets, primarily on teams that they believed would win those bets. That has also resulted in huge results.” However research is measured, it has to accommodate different approaches to realize the advantages of diversity, Spector said.

Failure is an important aspect of research. If there is no failure in research projects, then they are not at the right point on the risk-reward spectrum, said Spector. Rewarding failure may not seem like a good thing, but for research it can be essential. At Google, said Spector, “we view it as a badge of honor to agree that a certain line of advanced technology or research is not working and to stop and do something else. I think we need to have measurements like that in the world at large, although it’s clearly a challenging thing to do.”

Finally, the potential for serendipity needs to be rewarded. “If everything is so strongly controlled, I have a feeling we’ll do whatever the establishment feels is right and serendipity will be removed.” Serendipity often produces the creative disruption that reshapes entire industries, Spector concluded.