Introduction

I would like to consider the theme of this paper—the undergraduate research experience—in terms of four separate considerations: Does it make a difference? For whom? How do we know? What is the evidence of the last 10 years? I will begin with the answers to the first two questions. Yes, undergraduate research does make a difference—it makes a difference for students, for undergraduate curricula, for chemical science both as an academic discipline and as a profession and for the nation and society as a whole. But how do we know this? This question brings us to the main focus of this workshop—namely, assessment. I see my role in this forum as addressing not assessment of the value of research per se, but rather assessment of the value of the (undergraduate) research experience. Here, I will attempt to show how the first two questions posed have been answered (or can be answered), and by what means—namely, by a variety of assessment techniques applied during the past 10 years. These include data from the records of undergraduate research—site administrators in academia, survey responses, personal histories, and anecdotes, as well as the general impressions that most people involved with undergraduate education have formed over the years.

However, first of all, you may ask: "Why 10 years?" Suffice it to say that I chose that time frame because the National Science Foundation's REU (Research Experiences for Undergraduates) program got off the ground in 1986 with the first sites in place by 1987—hence this program, which specifically focuses on undergraduate research experiences, has been in operation for 10 years (see Box 7.1 for a short history). Now I do recognize that many undergraduates participate in research without REU funding. I picked REU because it represents a national recognition of the importance of undergraduate research and a commitment to provide research experience for undergraduates. In this sense, it marks a watershed or point of reference—and it started 10 years ago.

BOX 7.1

History Leading Up to the Research Experiences for Undergraduates Program. A bit of history may put the Research Experiences for Undergraduates(REU) program in perspective. After Sputnik made Americans focus again (more...)

Undergraduate Research Does Make a Difference

Does undergraduate research make a difference? To whom does it make a difference? From what point of view? I will look at this question successively from the point of view of the discipline, the profession, the nation, and the student. Let us begin with the discipline of chemistry.

If I had asked, for example, "Does organic chemistry (in the undergraduate curriculum) make a difference, and what is the evidence for that in the last 100 years?," you would, of course, think the question ludicrous and the answer obvious. The answer, I think, is equally obvious with respect to undergraduate research. Chemistry as an undergraduate discipline clearly has two components—one of content and one of method. It is a body of knowledge (complete with concepts, symbols, and facts) traditionally learned through the classroom lecture. It is also a collection of techniques (complete with observation, manipulation, data collection, and interpretation) traditionally learned in the laboratory. But chemistry is also about discovery and application —a dynamic synthesis of content and methodology to find out something, or make something, or do something new. This, of course, is what chemistry is all about, and what we call research.

So one way research experience might be said to "make a difference" is that it enhances, completes, or rounds out a science major's education. It helps develop an understanding of the complexity and context of science, not merely the content and methods. It is a way to validate the previous years of learning (analogous to the years of French grammar, vocabulary, dictation, and the reading of literature that can lead to communicating with others in French and writing one's own poem) and of contributing at last, in however small a way, to a body of knowledge that we call chemistry. What we have concluded here, then, is something I really believe should apply to all disciplines—but above all to those in the natural sciences: Research is a necessary component of a bachelor's degree education.

This is not a new idea—at least not in chemistry. Many undergraduate chemistry programs require undergraduate research; many others strongly recommend it and help make the opportunity available. This was also the conclusion arrived at by a group of NSF-REU chemistry site directors at a workshop held here in Washington in 1990, namely:

Chemistry is a dynamic experimental science for which research is an inherent component. Such a discipline requires "learning by doing," an inquiry approach, and an apprenticeship experience. A student's education in chemistry is incomplete without research experience.

Similar statements appear in a report issued by an analogous biological sciences site directors workshop held in 1993:

Research not only serves as an important adjunct to the traditional classroom, but is absolutely essential in modem, quality, undergraduate science education.

They also are found in the American Chemical Society's Committee on Professional Training guidelines for professional certification, and from the Council of Undergraduate Research.

Conclusion 1. Undergraduate research experience makes a difference in the discipline of chemistry because it belongs in the curriculum, just as a lecture course in organic chemistry or a laboratory course in methods of analysis does.

Of course, the finding of Conclusion 1 is not what the general public, or even many chemists, often mean when contemplating the "does it make a difference?" question. Instead, what they are thinking is: Does exposure to undergraduate research end up producing more and better trained chemists who will contribute to the economic health of the nation and the quality of life for its citizens? This, I am certain, is what Congress had in mind when it reenacted funding for undergraduate research experience in the mid-1980s. Here the questions are: Are we capturing people—that is, are more people majoring in chemistry? And are we retaining them so that they go on to work as chemists, or go on to graduate school, and then opt for research careers?

Let us look at one source of evidence relating to how undergraduate research bears on these issues. This information comes from records maintained for the past 10 years of undergraduate research and REU-site activity in chemistry at the University of Kansas. We have had the good fortune to be funded as a Chemistry NSF-REU site for the past 10 summers, and each year we bring in 10 to 12 students from other colleges who join about an equal number of our own undergraduates for 10 weeks of immersion in intensive and genuine research, including formal presentation of results.

Polling these students upon leaving the program and tracking them for subsequent years has generated the information summarized in Table 7.1. It is clear that a very significant number (at least 75 percent of the 216 students who have participated in our research programs since 1988) go on to advanced study in the chemical sciences, and another 4 percent go into chemistry-related jobs. If we compare these students with our majors in general, including those who do not participate in research, the sum of those two numbers is noticeably lower: closer to 50 percent. Similar results are reported at other schools—for example, by Professor John Hogg at Texas A&M: ''I estimate that up to 75 percent of students who participate in our NSF-REU program attend graduate school"—and at meetings and conferences.

TABLE 7.1

Postbaccalaureate Activity of Participants in the Chemistry Summer Undergraduate Research Experience at the University of Kansas from 1989 to 1997.

Conclusion 2. Undergraduate research experience makes a difference by increasing the numbers of individuals who choose careers in chemistry or who go to graduate school in the chemical sciences.

In addition to societal benefits accruing from a larger pool of talented people entering the chemical professions, we should also consider the various indirect benefits to society that might accrue from people educated in curricula involving research experience. For example, are more informed scientists and nonscientist citizens emerging as public school teachers, or as policy makers, communicators, journalists, science advocates and lobbyists, elected officials, critical thinkers, problem solvers, and simply citizens who are science literate? Although they certainly can be obtained, data bearing on these considerations are not as easy to come by. I have no special information in this regard to discuss today, as important as it might be for assessing the value of the undergraduate research experience.

Not unrelated to the concept of indirect benefits to society is the question of the inclusion of all of our citizenry. Does undergraduate research experience have an effect on the numbers of underrepresented people who become productive scientists or who join the ranks of the scientifically literate in this country? In other words, what about the impact of undergraduate research on minorities and women? My own data on the minority question are very limited, so I will give examples only as they relate to women. Our 10-year REU data show a real increase in the percentage of women applicants, such that it is now about 50 percent, and an increase in the percentage of women participants. Since research experience has been shown to have a positive influence on later graduate school and career choices (influence that seems not to be particularly gender specific), it follows that more women doing undergraduate research will lead to more women doing science later. Since the numbers are relatively small each year (10 to 12 students), there is a fair amount of scatter. Nevertheless, the trend is clear.

Conclusion 3. The last decade has shown a clear increase in the numbers of women participating in undergraduate research and attending graduate school.

Also, one might ask, has undergraduate research made a difference from the perspective of the student? For example, has it increased his or her understanding and appreciation of the major and of the discipline of chemistry? Has it helped in making career decisions? Has it helped in the decision to go to graduate school? Has it helped define an area of interest for future study or work? And, if a student did go to graduate school, has it made adjustment easier? Did it increase retention?

In 1995, I conducted a survey of current chemistry graduate students with the help of individuals from two other institutions, John Hogg at Texas A&M University and Dale Hawley of Kansas State University. A number of questions relating to those just posed were on the survey. There were 129 respondents, with 110 (85 percent) of those having engaged in research during the undergraduate years. There were 41 respondents from the University of Kansas (about 45 percent of the graduate students), 20 from Kansas State, and 70 from Texas A&M (about 25 percent of the graduate students). Details of the characteristics of those completing the survey forms are given in Box 7.2.

BOX 7.2

Characteristics of Students Completing the Three-School Survey. The students completing the three-school survey (the University of Kansas, Kansas State University, and Texas A&M University) were 61 percent male, 39 (more...)

Based on data obtained in the survey, where the data refer to those students who had participated in undergraduate research, the following can be observed:

• Less than 2 percent reported an unsatisfactory undergraduate research experience. We have no data as to numbers of students with such negative experience who did not go on to graduate school, but the data indicate that of the 85 percent of students in graduate school in our survey who had prior research experience, at least 97 percent had had a positive exposure.
•  For 77 percent of the students, undergraduate research was a contributing factor in the decision to go to graduate school. For 44 percent, it was the major factor.
•  Some 60 percent found that research experience helped in deciding which graduate school(s) to apply to.
•  An undergraduate research experience was seen as helpful in adjusting to graduate school (56 percent) and in getting started with graduate research (52 percent).
•  Almost 90 percent considered the experience useful in helping them decide on a general area of research to pursue in graduate study.
•  About two-thirds (67 percent) believed that prior research exposure helped them to "stay the course" and remain in graduate school.

Interestingly, the data showed no noticeable gender distinctions; that is, the responses from males and females were not significantly different. (The only instance where a difference was noticed was in the question about postdoctoral plans.)

Conclusion 4. Undergraduate research experience makes a difference for both men and women chemistry students by favorably disposing them toward graduate school, helping them select a school and area of research, better preparing them for graduate education, and influencing them to remain and complete a graduate degree.

We should also be concerned with questions having to do with other, more intangible benefits to the student resulting from undergraduate research experience. Such intangibles may relate to the establishment of mentor and peer relationships and of networks of advisors, co-workers, and friends. They may equally relate to the personal growth and increased self-confidence developing from an experience of common endeavor, and perhaps of real accomplishment and productivity. They surely should relate to the enhanced personal aspirations and broadened horizons that we believe may come from exposure to scholarly work, professionalism, dedicated scientists and motivated peers, and the joys and trials of discovery.

In this case, all the data we have are fairly anecdotal, coming from conversations or other communication with former students or their research advisors. I will illustrate by describing two individuals. Following his sophomore year at the University of Northern Iowa, the first student came as a 1988 summer REU participant to our department. He chose to work on an organic synthesis problem and, as he put it, "may have gotten a few drops of product." Nevertheless, he got a lot out of the program and still maintains close contact with most of the other participants who were with him back in 1988. After selecting a graduate school, because he remembered being captivated by one of our Friday seminar talks given by one of our faculty that described gas-phase mechanistic studies, the student made the important decision to go into gas-phase physical organic chemistry. He has just completed a postdoctoral stint, has developed methods that directly observe transition states, and has just started his academic career as an assistant professor at a major research university. This is an example of the profound influence of undergraduate research on a career path.

The second student is currently a junior at the University of Kansas and is a chemical engineering major, with an interest in biomedical engineering and a desire to attend medical school. She was a student in my organic chemistry class last year—the very top student out of a class that began with 450 others. She had never done research. Last summer, partly with my assistance, she located an internship working on a biochemistry project at the University of Cincinnati Medical School and returned, glowing with enthusiasm: "I know I want to do research." The director of the program there wrote me the following on July 31, 1997:

[She] has turned out to be exceptional, intellectually and technically. Considering that she had no background in gene transcription and had never held a pipette, [her summer research advisor] is amazed. On Thursday, she will present the results of her research at a minisymposium.

For both of these people, there can be little doubt as to the value of the undergraduate research experience. Further evidence comes from written comments provided by the students who completed the three-school survey described above, just a few of which I will take the time to read now.

I participated in research at a major pharmaceutical firm, at a different university for a summer, and worked in a lab at my university for 3 years. I consider the experience gained to be the most important part of my undergraduate experience.

Even though I did some research at my small college prior to doing research at a big university, I had no idea what "real" research was like. The research work at the big university influenced more than 95 percent of my decision.

My undergraduate research helped to integrate me with the chemistry faculty and resulted in 2 full papers which have since appeared in peer-reviewed journals.

(You will note that I have not addressed research output or productivity, although we seem to average about 0.5 papers per student in the program.) Another comment from a student:

I must stress the importance of the research experience in making students an integral part of the department and converting them from bench warmers in class to productive, thinking scientists.

A research experience is critical to the decision process of selecting a division of study, and helping to decide whether a final goal of academic or industrial [work], teaching, or research is desired.

Conclusion 5. Personal testimony, informal communication, written comments and responses to surveys, and the experience of mentors and advisors all indicate that undergraduate research experience makes a positive difference in the education and the professional and personal development of its participants.

Summary

A variety of assessment methods, some more quantitative than others, can be applied to the question of determining the value of the undergraduate research experience. Results obtained and discussed today indicate that such experience provides an essential component of science education, contributes to more informed career decisions, generates more students choosing careers in chemistry, promotes success in graduate programs and probably future work in chemistry, and results in increased personal confidence and a better appreciation for and understanding of science. Furthermore, it contributes to the nation's research enterprise by playing an important role in producing more, better prepared, and better qualified scientists for the future.

Discussion

Beverly K. Hartline, Office of Science and Technology Policy: I definitely share the enthusiasm for the value of undergraduate research experiences. But I am very curious to know if there are any data on the performance of these students in courses subsequent to having had that experience as compared with students who have not had the experience (or even to their own previous performance). Is information available on their actual performance in graduate school? In industry, if they take an industrial job? That is, are they really better prepared? Or has their intuition been improved as a result of the undergraduate research experience.

K. Barbara Schowen: Those are very important questions. I personally don't have any data on how students subsequently perform in industry, and so I can't address that point. Do they do better in their returning courses? I also have no hard data here. However, they return from the experience very enthusiastic about their discipline and are certainly going to be paying attention to their courses. But I can't answer the question, as to whether they earn more "A"s than they did before. I think it is possible to obtain these data. In fact, one of the points of this workshop is to determine what metrics should be used in assessing the value of the undergraduate research experience and how to obtain the needed data.

Michael P. Doyle, Research Corporation: Before 1982, the Associated Colleges of the Midwest (which included Carleton, Grinnell, St. Olaf, and Macalester) carried out a similar survey of their undergraduate research participants, and about 650 people responded. Curiously, the same number, about 70 percent, said it was an important determinant of their future career and their success in that career. This survey focused on people who had been out of the program for 5 or more years, and so they had been through school. The data seem to confirm what we have known for a long time.

In your particular case, participation in the Research Experiences for Undergraduates program has a special requirement—namely, that the students have already professed an interest in chemistry, are entering a chemistry program, and are predisposed to go on to graduate school. So, in fact, aren't you just measuring what you expect to find?

K. Barbara Schowen: That is a hard question to answer. The control experiment would be to poll all the people who had not participated in research and find out what percentage of them went to graduate school.

Raymond E. Fornes, North Carolina State University: Many of you are aware that there is a national conference on undergraduate research each year. If you look at the data on students who have participated in that conference (and I think there are on the order of 2,000 papers presented at the national meeting each year), you find that there is a much higher percentage coming from the small colleges—the Carletons, for example—than from the major research universities.

If I understood you correctly, in the REU program, you got five students per year from Kansas and five from other colleges.

K. Barbara Schowen: No, we had 10 to 12 from colleges other than our own. We have about 25 total in the program.

Raymond E. Fornes: But still, that is on the order of 10 percent of your undergraduate student population in chemistry. So the question is, how do you foster greater participation in research experiences, particularly in the large universities, where such a high percentage of students are in the sciences?

K. Barbara Schowen: That is one thing that I have spent a lot of time doing as an undergraduate advisor in our department. We strongly encourage our students to participate in research, and our department tries to make this happen. Professor Daryle Busch, who is also at the workshop, has a large number of undergraduates working in his group.

From the data that came from the three-school survey, about 45 percent of the students were from large public institutions, and the others were from small 4-year colleges. I believe that the information gathered from the two groups was similar.