6Assessment and Evaluation: What Can Be Measured in Mentorship, and How?

Process-Oriented Model

In the ideal case, measurement models would map onto theoretical models to test research questions and hypotheses of mentorship processes and outcomes. In the committee's review of assessment methods, recent theoretical and empirical evidence supports a process-oriented model of mentorship (Figure 6-1) (Eby et al., 2013) that can be mapped onto assessment methodologies. This model holds that personal, contextual, and relational inputs shape the characteristics of the mentoring relationship processes, and these relationship processes influence cognitive, emotional, and behavioral outputs. Outputs from mentorship in STEMM contexts vary widely across the literature, with examples including psychological processes such as self-efficacy, learning or skill, scholarly achievement, and enhanced career aspirations and advancement including persistence in STEMM pathways (Crisp and Cruz, 2009; Eby et al., 2013; Gershenfeld, 2014; Ghosh, 2014; Ghosh and Reio, 2013; Jacobi, 1991; Pfund et al., 2016; Sadler et al., 2010; Syed et al., 2011).

According to this process model, mentorship includes active functions such as career support or instrumental support (i.e., sponsorship, coaching, exposure and visibility, protection, and challenging work assignments) and psychosocial support (i.e., acceptance, counseling, and friendship) that were discussed in Chapter 2. Additional roles include passive functions, such as role modeling,⁹ in which a mentor serves as an inspirational example of the norms, attitudes, and behaviors necessary to achieve success (Lockwood and Kunda, 1997). Mentorship also includes negative experiences, including mismatch within the mentorship dyad, distancing behavior, manipulative behavior, lack of expertise, and general dysfunctionality, as discussed in Chapter 5 (Eby et al., 2004; Eby, Durley et al., 2008; Kram, 1985b). Benevolent mentorship support functions—career support, psychosocial support, and role modeling, for example—promote relationship quality, which includes overall relationship satisfaction, trust, reciprocity, and effectiveness (Kram, 1985b), whereas negative experiences diminish relationship quality. Relationship quality, in turn, reciprocally influences future levels of provided and received mentor support functions (Eby et al., 2013). For research mentoring in STEMM, performance outputs encompass an array of research skills, as well as critical research products such as publications.

A landscape review conducted for this report identified 35 assessments of mentoring relationship processes in postsecondary educational STEMM contexts from the perspectives of mentees, mentors, or programs/institutions, many of which contain components that map onto process models (Hernandez, 2018). Most of these assessments have focused on measuring characteristics of the mentoring relationship from the mentee's perspective, and the majority of those assessments focused on undergraduate and graduate students, with fewer looking at postdoctoral researchers. Of the few assessments focused on the mentor's perspective of the mentoring relationship, most examined university faculty, graduate student, and postdoctoral researcher perceptions of the mentoring relationship they had with undergraduate mentees (Hernandez, 2018). Assessments of mentoring relationships from the program or institutional perspective drew on the perceptions of institutional staff members who run mentorship programs or faculty mentors involved in those programs.

The quantity and quality of validity evidence varies substantially across mentee, mentor, program, and institutional evaluation perspectives and within specific assessments from each perspective. Figure 6-2 summarizes the strength of the validity evidence based on assessment content, internal structure, and relationships among processes within the process-oriented model of mentorship (Eby et al., 2013). Table 6-1 lists the instruments that have moderate levels of validity evidence supporting their use (Hernandez, 2018).

FIGURE 6-2

Synthesis of mentoring relationship processes validation evidence in postsecondary STEMM contexts. NOTES: Solid lines separate institutional perspective from mentor/mentee perspective, as there is little to no evidence connecting these perspectives. For (more...)

Assessments from the mentee perspective examined types of career and psychosocial support mentees received as well as overall mentor relationship quality. Items in these assessments ranged from general support functions that apply across contexts, such as goal setting, to support functions that are specific to STEMM contexts, such as research collaboration. Assessments from the mentor perspective examine a variety of behaviors categorized as provision of career support and psychosocial support. Assessments at the program or institution level included items that ranged from general support functions to items that are specific to STEMM contexts, such as fostering research independence.

TABLE 6-1

Assessments by Career Stage with Moderate Levels of Validity Evidence.

Outcomes of Mentorship in STEMM for Mentees

For undergraduates in STEMM, participating in mentored research experiences has been linked to self-reported gains in research skills, productivity, and retention in STEMM (Laursen et al., 2010; Linn et al., 2015; Sadler and McKinney, 2010). Studies have also shown that research experiences combined with quality mentorship that includes providing psychosocial and career support and networking opportunities contributes to students feeling integrated into STEMM fields (Estrada et al., 2018). Effective mentoring relationships have been shown to influence undergraduate mentees' confidence in their research skills, a key predictor of persistence in STEMM (Byars-Winston et al., 2015). One investigator described STEMM environments as ideal for the development of undergraduate mentor-mentee relationships because there is often a focus on working in laboratories (DeAngelo, 2016), which places the faculty member and student in a one-on-one situation conducive to mentorship. Still, students and faculty have to initiate this pairing on their own.

As noted in a 2017 National Academies report on undergraduate research experiences in STEM (NASEM, 2017b), mentees perceive mentors who model ethical behaviors, kindness, and competence as exhibiting outstanding mentor qualities (Johnson, 2002; Mullen et al., 2000; Rice and Brown, 1990). In addition, research has shown that perceived mentor effectiveness indirectly predicts enrollment in science-related doctoral or medical degree programs (Byars-Winston et al., 2015).

Graduate students who have good mentoring relationships are more likely to persist in their academic decisions (McGee and Keller, 2007; Williams et al., 2016a), with positive mentorship cited as the most important factor in completing a science, technology, engineering, and mathematics (STEM) degree (Ashtiani and Feliciano, 2012; Solorzano and Yosso, 2000). Quality mentorship focusing on graduate students' psychosocial needs appears to increase how mentees perceive the quality of the mentoring relationship and how satisfied they are with that relationship, which in turn enables them to see themselves as more competent STEMM researchers (Tenenbaum et al., 2001; Waldeck et al., 1997). Mentored graduate students and medical trainees are also more likely to publish their research than those who are not mentored (Steiner et al., 2002, 2004; Wingard et al., 2004).

The association between quality mentoring relationships and achievement among mentees from groups who are underrepresented (UR) in STEMM¹² is even stronger (NASEM, 2017a).¹³ Evidence suggests that positive mentor-mentee relationships and quality mentorship are particularly important for integrating women and UR students into the STEMM academic community (Anderson and Kim, 2006; Byars-Winston et al., 2015; Estrada et al., 2018; Felder, 2010; Good et al., 2000; Griffith, 2010; Huang et al., 2000; Lewis et al., 2016; Lisberg and Woods, 2018). Studies have also shown that quality mentorship increases recruitment of UR mentees into graduate school and research-related career paths (Hathaway et al., 2002; Junge et al., 2010; Nagda et al., 1998; Thiry and Laursen, 2011).

Outcomes of Mentorship in Higher Education Outside of STEMM for Mentees

Researchers have conducted a wide range of qualitative and quantitative studies on mentorship outcomes in higher education outside of STEMM. In qualitative studies, for example, investigators used case study methods and interviews to study recommended characteristics of mentorship, how students and mentors experience the mentoring relationship, and what both students and mentors expect from mentoring relationships and what their roles are in that relationship (Baker and Griffin, 2010; Bell and Treleaven, 2011; Griffin, 2013). For the most part, quantitative research has examined college adjustment (Apprey et al., 2014), career and personal development (Haddock et al., 2013; Kinkel, 2011; Sams et al., 2015), and measures of academic progress and success (Fox et al., 2010; Hu and Ma, 2010; Zell, 2011).

Most of these studies in higher education outside of STEMM did not distinguish between mentorship and other forms of supportive relationships, including those with advisors, institutional agents, developers, and coaches (Baker and Griffin, 2010; Bettinger and Baker, 2011; Museus and Neville, 2012; Tovar, 2015). Nonetheless, there are lessons from these studies that suggest what outcomes STEMM mentees might experience. This research suggests, for example, that informal mentorships are more likely to be successful for mentees and result in outcomes superior than with formal mentorship, which is when relationships are based on assigning students to mentors (Davidson and Foster-Johnson, 2001; Gandara and Maxwell-Jolly, 1999).¹⁴ This research also shows that career and psychosocial support in mentorship often contribute in different ways to different types of outcomes for mentees, and that career support typically results in better career outcomes, such as greater publication output for graduate students (Haeger and Fresquez, 2016; Tenenbaum et al., 2001). Psychosocial support results in outcomes that are crucial for student well-being and other criteria necessary for promotion and productivity, such as greater satisfaction with the mentoring relationship and commitment to one's own academic program (Phinney et al., 2011). Other positive outcomes from mentorship programs include increased academic performance and involvement in programs at the college or university (Brittian et al., 2009; Dahlvig, 2010), better transition and adjustment to the college environment (Smojver Ažić and Antulić, 2013), improved personal and career development (Kinkel, 2011), more degrees conferred and persistence through programs (Gross et al., 2015), and positive civic outcomes such as increased social responsibility and socially responsive leadership (Haddock et al., 2013).

Outcomes from a Relationship Perspective

Mentoring relationships can be characterized by the purpose, intensity, and duration of the relationship. Successful mentoring relationships result from a mentor's intentional and purposeful commitment to helping the mentee succeed (Baker and Griffin, 2010). Additionally, mentorship may help develop students' time management skills, study skills, communication skills, and other transferable skills sets, as well as helping them adjust to college (Michael et al., 2010; Salinitri, 2005). Helping to guide and engage students in research, providing direction in career goals, and creating a sense of belonging in college departments are strategies that have proved successful in mentorship programs (Crisp et al., 2017).

Measuring outcomes from the mentoring relationship perspective highlights the value of having parallel measures from both sides of the relationship: that of the mentor and the mentee. Such parallel measures can elucidate the degree to which mentees and mentors have shared views about the mentoring relationship and mentoring activities, which can be an indicator of their working alliance. One example of parallel mentoring relationship measures is from the Howard Hughes Medical Institute Gilliam Fellowships for Advanced Study, a predoctoral program for UR students in STEMM. A survey posed questions to Gilliam mentors and mentees in dyadic pairs about behaviors in the mentoring relationships related to facilitating students' research and career development and science identity. Results of the survey revealed a mismatch on some aspects in the mentoring relationship. Namely, mentors reported displaying more of the desired behavior, such as, mentors sharing their own research career pathway, highlighting and giving direction for improving mentees' research outcomes, and affirming mentees' ability to be a scientist, than their mentees reported perceiving (see Table 6-2) (Pfund et al., 2019). These findings indicate that further inquiry into how different views of mentoring activities influence mentorship outcomes could be useful, and they also point to the potential value of mentorship education in supporting mentors' career facilitation for students.

Few studies on mentorship outcomes appear to use theoretical frameworks focused on the relational elements of mentoring, such as social support, that emphasize how relationships reduce stress and promote coping, or developmental support, which links mentorship to the college student developmental process. However, several studies (Aikens et al., 2016, 2017) have used social capital theory as a framework for examining the effect of mentorship structures between students, doctoral and postdoctoral scholars, and faculty on various outcomes.¹⁵ These investigations found that in “closed mentorship triads,” which included a faculty mentor, a graduate student or postdoctoral mentor, and an undergraduate student mentee,¹⁶ interactions were the most beneficial for mentee outcomes such as science identity development (Aikens et al., 2016, 2017), scholarly productivity, and intentions to pursue a STEM Ph.D. (Aikens et al., 2016). In addition, several researchers have developed parallel measures for mentors and mentees in STEMM based on social cognitive career theory and science identity as well as multicultural theory (Byars-Winston et al., 2016).¹⁷ These parallel mentor and mentee measures assess elements in the mentoring relationship related to mentees' research self-efficacy beliefs and mentors' cultural diversity awareness (see Table 6-3).

TABLE 6-2

Results from a Paired Survey of Mentors-Mentees in the Howard Hughes Medical Institute Gilliam Fellowships for Advanced Study Program.

TABLE 6-3

Parallel Mentor and Mentee Measures Assessing Social Cognitive Career Theory (SCCT) Variables and Cultural Diversity Awareness of Mentors.

Measuring Mentor Motivations and Correlates

Assessment and measurement of mentorship could integrate how and why mentors participate in mentorship and what they gain from successful mentorship. For example, one qualitative case study found that graduate students and postdoctoral researchers who mentored undergraduates in research reported improved career preparation and qualifications, cognitive and socioemotional growth, improved teaching and communication skills, greater enjoyment of their own apprenticeship experience, and twice as many benefits as challenges (Dolan and Johnson, 2009). Their motivations for engaging in mentorship were largely about how mentorship would serve as a means to an end, though the benefits and challenges they reported indicated a longer-term vision of how mentorship influenced their personal, cognitive, and professional growth. At the same time, some mentors in this study reported that mentorship of undergraduates made their work lives more enjoyable while generating emotional costs. Several investigators have reported that mentors benefit from a sense of personal fulfillment through knowledge and skill sharing, honing their leadership skills, career preparation, and cognitive growth (Dolan and Johnson, 2009; Eagan et al., 2013; Laursen et al., 2010).

Another qualitative study determined that mentors had both career and intrinsic motivations for mentorship in the context of undergraduate research, which appeared to differ by career stage (Hayward et al., 2017). Career motivators for faculty included increased productivity, help in recruiting future students, increased prestige for the university resulting from students presenting at conferences, and helping prepare students for graduate work and careers. Intrinsic motivators included improved teaching and mentorship skills, feelings of doing something positive, preparing future scientists, and increased energy and enthusiasm in the lab. Faulty mentors of undergraduates were motivated by their belief that mentorship informed their teaching and added fun and enthusiasm to their work, while negative factors included the need for additional time, effort, and funding; increased tension; increased difficulty of gauging students' research ability; and little recognition or reward (Dolan and Johnson, 2010). Forming mentoring relationships with graduate students helped faculty recruit undergraduates and gain a better sense of postgraduates, but study participants had trouble gauging the effectiveness of mentorship.

Research outside of STEMM indicates that mentors' commitment to the mentoring relationship matters for mentorship outcomes (Allen and Eby, 2008). Given competing role demands on mentors and mentees in STEMM and work settings, mentor commitment is not necessarily a given and is often an outcome of many factors (Aryee et al., 1996). In fact, research outside of STEMM indicates that mentors' identities and their perceptions of the benefits of mentorship toward their own career goal progression play a role, along with factors such as altruism and the presence of effective schemata for developing and sustaining relationships with mentees (Ragins, 2009).

Even though effective mentorship has been shown to relate to positive career outcomes for mentors in workplace settings (Ghosh and Reio, 2013; Rogers et al., 2016), the relationships between effective mentorship and career outcomes for mentors in STEMM settings are not always self-evident. Research on work performance (Kerr, 1995; Van Eerde and Thierry, 1996) suggests that individuals have to understand that certain tasks and the quality of task completion will factor into organizational reward systems and ultimately the individual-level compensation and rewards they receive. In other words, the value and attention paid to mentorship quality might change if it became a tracked and managed component of universities' and research organizations' performance appraisal system for faculty and other researchers who engage in STEMM mentorship (Aryee et al., 1996).¹⁸ It is important to note that there may be unintended consequences of efforts to track and manage mentorship, especially if mechanisms are not carefully identified and vetted by professional assessment developers to minimize inequities and bias.

Dyadic Data Analysis

Relatively recent advances in statistical methodology now allow for characterizing reciprocal relationships through dyadic data analysis (Kenny, 1994; Kenny et al., 2006).¹⁹ Dyadic data analysis involves collecting data from both the mentor and the mentee over time to reveal how the perceptions and experiences of each influence the other. For example, a mentor's perceptions of the mentee have the potential to influence the mentee's self-perceptions, but this influence can only be examined if data are collected from both the mentor and the mentee over time. This methodology allows researchers to investigate dynamic feedback loops between mentor and mentee, where each informs the other regarding what is or is not needed from the relationship, how the relationship quality and characteristics such as trust development shift over time, and how this influences both mentor and mentee. For instance, this method could reveal how change in trust over time from both the mentor and the mentee perspective influences mentee perceptions that they are receiving psychosocial support or mentee confidence in their ability to be successful in a STEMM career. One study used a dyadic approach to characterize reciprocal feedback between mentors and mentees in a STEMM research experience context (Griese et al., 2016).

Social Network Analysis of Mentorship

Advancements in mentorship theory point to the importance of networks of mentoring relationships, particularly for individuals from historically UR groups (Downing et al., 2005; Glessmer et al., 2012; Higgins, 2000; Higgins and Kram, 2001; Higgins and Thomas, 2001; Packard, 2003a; Packard et al., 2004). Recent advancements in measurement and statistical methodology now allow researchers to capture and quantify characteristics of mentorship-related networks as social networks (Scott, 2017). Social networks can be conceptualized either as “whole networks” or as “ego networks.”

Whole networks are systems such as a mentorship group. Whole network analysis could be used to analyze the value of collective or group mentorship, including the value of the network based on the resources offered by its members, such as expertise and information; the diversity of its members; which relationships within the network are most influential; how interconnected members must be for the network to be valuable to its members; where there might be gaps in the network; and which members of the network serve as hubs for information or resources such as high-quality feedback. Several researchers have begun to measure and categorize beneficial triadic mentor network structures as the simplest form of a whole network (Aikens et al., 2016, 2017; Morales et al., 2018) and to identify and characterize successful mentorship communities (Chariker et al., 2017), but much more can be done to determine how mentorship networks operate and their distinctive impact and value.²⁰

Ego networks are the connections, or lack thereof, of a single individual and the resources available, or not, to the individual through their connections. Ego network analysis could be used to examine the mentorship resources available to a given mentee and how these resources relate to their personal characteristics and outcomes. For example, mentees with different racial, ethnic, and gender identities can differ in their mentorship networks in ways that may or may not influence their outcomes (Aikens et al., 2017). Longitudinal ego network research is appropriate to determine whether mentees with different personal characteristics are more or less likely to develop mentorship networks that meet their needs. For instance, mentees who identify strongly with their mentor may perceive that they are receiving both career and psychosocial support and thus may require a simpler dyadic mentorship structure to meet their needs. Mentees who do not identify strongly with their mentor either personally or professionally may benefit from a more elaborate network of mentors, including others who share their identities or particular career interests. These questions could be addressed through systematic analysis of the ego networks of mentees related to their personal characteristics and outcomes.

Further Questions for Inquiry

Understanding the mechanisms by which mentorship is initiated, developed, and sustained, and if they are effectual, is important for theory building and for practical purposes. For example, if research can identify specific, favorable mentee and mentor behaviors, it could be possible to enhance and encourage those behaviors through programming and evaluation systems, and thus improve mentoring relationships and resulting outcomes.

Ideally, assessments would identify important milestones in developing mentoring relationships. In addition, assessments could provide details on whether relationships develop in a linear manner or if there are discontinuous changes or time-bounded needs of mentees, mentors, or mentoring relationships that must be taken into account to develop an effective mentoring relationship and fully realize its benefits to mentees, mentors, and the STEMM enterprise. For more quantitative data, statistical techniques could be used to identify unobservable subgroups based on measured variables or trends in larger data sets, such as probability-based latent class analysis (Bauer and Shanahan, 2007; Oberski, 2016; Pastor et al., 2007; Wachsmuth et al., 2017). More work is needed to minimize selection bias in assessing mentoring outcomes, for example, matched control groups or propensity score matching.

Most mentorship theories suggest that mentoring relationships change over time,²¹ and most correlational research assumes that change is linear—that as trust increases, for example, so does relationship quality. However, experience implies that relationships can shift suddenly, such as when one act of betrayal irreversibly destroys a relationship or when one act of kindness transforms a struggling relationship. Research on turning points in close relationships suggests using both quantitative and qualitative methods to develop robust, explanatory theory. Research could potentially determine if there are predictable patterns of discontinuous change, identify experiences that fundamentally alter mentoring relationships, and learn if positive turning points can repair a previously damaged mentoring relationship (NRC, 2002; Warfa, 2016).

There has been little research on multilevel influences arising from mentoring relationships being nested within workgroups, academic departments, research laboratories, organizations including colleges and universities, and industries or academic disciplines. Research is also lacking on aggregate effects that go beyond the individual, such as work-group- or department-level effects. Multilevel modeling can help examine individual, dyadic, group, and organizational effects on the mentoring relationship.