The Impact of Informal Mentorship in Academic Collaborations

TL;DR

This study analyzes the effects of informal mentorship in scientific collaborations, demonstrating its positive impact on research impact and exploring gender dynamics, across multiple disciplines and over a century.

Contribution

It provides the first large-scale causal analysis of informal mentorship effects on scientific impact and gender-related citation dynamics in academia.

Findings

Mentorship quality causally increases paper impact.

Impact grows with the number of mentors and mentor experience up to 30 years.

Gender dynamics influence citation gains and mentorship effectiveness.

Abstract

Inspired by the numerous benefits of mentorship in academia, we study "informal mentorship" in scientific collaborations, whereby a junior scientist is supported by multiple senior collaborators, without them necessarily having any formal supervisory roles. To this end, we analyze 2.5 million unique pairs of mentor-prot\'eg\'es spanning 9 disciplines and over a century of research, and we show that mentorship quality has a causal effect on the scientific impact of the papers written by the prot\'eg\'e post mentorship. This effect increases with the number of mentors, and persists over time, across disciplines and university ranks. The effect also increases with the academic age of the mentors until they reach 30 years of experience, after which it starts to decrease. Furthermore, we study how the gender of both the mentors and their prot\'eg\'e affect not only the impact of the prot\'eg\'e post mentorship, but also the citation gain of the mentors during the mentorship experience with their prot\'eg\'e. We find that increasing the proportion of female mentors decreases the impact of the prot\'eg\'e, while also compromising the gain of female mentors. While current policies that have been encouraging junior females to be mentored by senior females have been instrumental in retaining women in science, our findings suggest that the impact of women who remain in academia may increase by encouraging opposite-gender mentorships instead.

Full text

Introduction

Mentorship contributes to the advancement of individual careers [1, 2, 3] and provides continuity in organizations [4, 5]. By mentoring novices, senior members pass on the organizational culture, best practices, and inner workings of a profession. In this way, the mentor-protégé relationship provides the human glue that links generations within a field. Mentorship can also alleviate the barriers of entry for underrepresented minorities, such as women and people of color, thereby acting as an equalizing force [6, 7, 8, 9]. Most workplaces have shifted from the classic master-apprentice model towards a team-based model where the mentorship of junior members is distributed amongst the senior members of the team. As a result, informal mentorship, whereby juniors receive support from senior colleagues without a formal supervisory role, has become commonplace [10, 11]. Being informal makes such mentorship invisible and thus elusive to study.

Academia has seen a similar shift towards a team-based model [12] which makes it an ideal application example to study informal mentorship. It has already proven to be an effective test bed to study a wide variety of topics, including team assembly [13], the role of past experience [14], individual productivity [15, 16, 17], diversity [18], and innovation [19], thereby giving rise to the field of Science of Science [20]. Importantly, academic papers provide a documented record of millions of collaborations spread over decades, along with an established measure of success, namely citation count. Motivated by this observation, our study of informal mentorship focuses on academic collaborations between junior and senior scientists, since such collaborations play an important role in shaping the junior scientist’s persona, both in terms of their research focus [21], professional ethics and work culture [22]. Furthermore, inspired by the expanding literature on gender equity and diversity in science [23, 24, 25, 26], we analyze the mentorship experiences from the perspective of both female and male scientists.

Compared to previous studies on mentorship in academia [27, 28, 29, 30, 31, 32, 33, 34], ours has the following advantages. First, we capture multiple informal mentor-protégé relationships per collaboration, rather than restricting our analysis to formal student-advisor relationships. Second, we avoid sample selectivity as well as recall and recency biases, since we analyze the actual scientific impact of collaborations rather than self-reported information. Third, we analyze thousands of journals spanning multiple scientific disciplines, rather than focusing on just a single journal or discipline. Fourth, we construct careful comparisons between millions of mentor-protégé pairs, allowing us to establish and quantify the causal effect of the mentorship experience on scientific careers. Finally, our study complements the literature that studies the impact of mentorship on attrition from science [35], as we consider protégés who remain scientifically active after the completion of their mentorship period.

Results

We analyze 115 million scientists and 130 million papers gleaned from the Microsoft Academic Graph (MAG) dataset [36], which contains detailed records of scientific publications and their citation network. In addition to MAG we use other external data-generating techniques and sources to establish the gender of scientists and the rank of their affiliations (see Methods).

We distinguish between junior and senior scientists based on their academic age, measured by the number of years since their first publication. The junior years are those during which a scientist participates in graduate and postdoctoral training, and possibly the first few years of being a faculty member or researcher. In contrast, the senior years are those during which a scientist typically accumulates experience as a PI and transitions into a supervisory role. For any given scientist, we consider the first 7 years of their career to be their junior years, and the ones after that to be their senior years. Whenever a junior scientist publishes a paper with a senior scientist, we consider the former to be a protégé, and the latter to be a mentor, as long as they share the same discipline and US-based affiliation. Our use sample consists of 2.5 million unique mentor-protégé pairs, spanning nine disciplines (Biology, Chemistry, Computer Science, Economics, Engineering, Geology, Mathematics, Medicine, Psychology) and over a century of research.

We consider two alternative measures of mentorship quality. The first is the average impact of mentors prior to mentorship, where the prior impact of each mentor is computed as their average number of citations per annum up to the year of their first publication with the protégé. This reflects the success of mentors and their standing and reputation in their respective scientific communities. We refer to this measure as the big-shot experience, as it captures how much of a “big-shot” the mentors of the protégé are. The second measure of mentorship quality that we consider is the average degree of mentors prior to mentorship, where the degree of each mentor is calculated in the network of scientific collaborations up to the year of their first publication with the protégé. We refer to this measure as the hub experience, as it reflects how much of a “hub” each mentor is in the collaboration network. These two measures of mentorship experience take the role of independent variables in our study.

Having discussed our measures of mentorship quality, we now discuss the mentorship outcome, which we conceptualize as the impact of the protégé during their senior years without their mentors. We measure this outcome by calculating the average impact of all the papers that the protégé published post mentorship without their mentors. The impact of each such paper is calculated as the number of citations that it accumulated five years post publication, denoted by $c_{5}$ [18]. Such an outcome measure allows us to assess the quality of the scholar that the protégé has become after the mentorship period has concluded.

We aim to establish whether mentorship quality (measured by big-shot experience or hub experience) has a causal effect on post mentorship outcomes, and we aim to quantify this effect. To this end, we use coarsened exact matching (CEM), a technique widely used when estimating causal effects in observational studies [37]. Intuitively, CEM allows us to select a group of protégés who received a certain level of mentorship quality (treatment group), and match it to another group of protégés who received a lower level of mentorship quality (control group). Comparing the outcome of the two groups allows us to estimate the average causal effect. In more detail, for each measure of mentorship quality, we create a separate CEM where the treatment and control groups differ in terms of that measure, but resemble each other in terms of all of the following: the number of mentors, the year in which the protégé published their first mentored paper, the scientific discipline of the protégé, the gender of the protégé, the rank of the affiliation of the protégé on their first mentored publication, the number of years starting from the first year post mentorship until the protégé’s last active year, and the average academic age of the mentors, which is computed based on the academic age of each mentor in the year of their first publication with the protégé. Finally, when studying the impact of the big-shot experience, we make sure that the two groups are similar in terms of the hub experience, and vice versa.

For every independent variable, be it big-shot experience or hub experience, let $Q_{i}$ denote the $i^{\textnormal{th}}$ quintile of the distribution of that variable. Then, for $i\in\{1,2,3,4\}$, we build a separate CEM where the control and treatment groups are $Q_{i}$ and $Q_{i+1}$, respectively. The CEM results are depicted in Fig. 1. As can be seen, the big-shot experience increases the post-mentorship impact of protégés by up to 36%; we refer to this as the big-shot effect. The hub experience also increases the post-mentorship impact of protégés, although the increase never exceeds 7%; we refer to this as the hub effect. Clearly, regardless of the choice of control and treatment groups, the big-shot effect is always significantly greater than the hub effect. As such, we restrict our attention to the big-shot effect throughout the remainder of our study.

In Fig. 2, we take a closer look at the big-shot effect, to understand how it is affected by the year in which the mentorship started, the age of the mentors, and the number of mentors. In particular, Fig. 2a analyzes the big-shot effect across the years of the protégé’s first publication. As can be seen, the big-shot effect is increasing over the years, and nearly doubled in the past two decades. On the other hand, Fig. 2b analyzes the big-shot effect across varying ages of the mentors. This shows that, as the mentors get older, their impact on the protégé continues to increase until their academic age exceeds 30 years, after which the impact starts to decrease. In Fig. 2c, we analyze the big-shot effect given varying numbers of mentors. This shows that the impact increases with the number of mentors, and that the protégé benefits from having more than 5 mentors during the mentorship period. Finally, in Supplementary Figures S1, S2 and S3 we show that the big-shot effect persists regardless of the discipline, the university rank, and the gender of the protégé.

Next, we turn to an exploratory analysis where we investigate the post-mentorship impact of protégés while taking into consideration their gender as well as the gender of their mentors. To this end, let $F_{i}$ denote the set of protégés that have exactly $i$ female mentors. We take the protégés in $F_{0}$ as our baseline, and match them to those in $F_{i}$ for some $i>0$, while controlling for the protégé’s big-shot experience, number of mentors, gender, discipline, affiliation rank, and the year in which they published their first mentored paper. Then, for any given $i>0$, we compute the change in the post-mentorship impact of the protégés in $F_{i}$ relative to the post-mentorship impact of those in $F_{0}$; we refer to this comparison by writing $F_{0}$ vs. $F_{i}$. The outcomes of these comparisons are depicted for male protégés in Fig. 3a, and for female protégés in Fig. 3b. As shown in this figure, having more female mentors tends to decrease the mentorship outcome, and this decrease can reach as high as 35%, depending on the number of mentors and the proportion of female mentors.

So far in our analysis, we only considered the outcome of the protégés. However, mentors have also been shown to benefit from the mentorship experience [1]. With this in mind, we measure the gain of a mentor from a particular protégé as the average impact, $\langle c_{5}\rangle$, of the papers they wrote with that protégé during the mentorship period. We compare the average gain of a female mentor, $F$, against that of a male mentor, $M$, when mentoring either a female protégé, $f$, or a male protégé, $m$. More specifically, we compare mentor-protégé relationships of the type $(f,F)$ to those of the type $(m,F)$, where $f$ and $m$ are matched based on their discipline, affiliation rank, number of mentors, and the year in which they published their first mentored paper. Similarly, we compare relationships of the type $(f,M)$ to those of the type $(m,M)$, where $f$ and $m$ are matched as above. The results of these comparisons are presented in Fig. 3c. In particular, the figure depicts the gain from mentoring a female protégé relative to that of mentoring a male protégé; these results are presented for female mentors and male mentors separately. As can be seen, by mentoring female instead of male protégés, the female mentors compromise their gain from mentorship, and suffer on average a loss of 18% in citations on their mentored papers. As for male mentors, their gain does not appear to be significantly affected by taking female instead of male protégés.

In this paper, we studied informal mentorship in academia, whereby a junior scientist receives support from multiple senior collaborators without necessarily having a formal supervisory relationship. Having conceptualized mentorship quality in two ways—the big-shot experience and the hub experience—we found that both have an independent causal effect on the protégé’s impact post mentorship without their mentors. Interestingly, the big-shot effect is larger than the hub effect, implying that the scientific impact of mentors matters more than their number of collaborators. We further analyzed the big-shot effect to understand how it is affected by the number of mentors, the age of the mentors, and the year in which the mentorship started. This analysis revealed that the effect of mentorship increases with the number of mentors, and nearly doubled over the past two decades. Furthermore, as the mentors get older, their impact on the protégé increases until they reach 30 year of experience, after which their impact decreases. We also showed that the big-shot effect persists regardless of the discipline, the university rank, and the gender of the protégé. Finally, we studied how the gender of both the mentors and their protégé affects not only the impact of the protégé, but also the gain of the mentors, measured by the citations of the papers they published with the protégé during the mentorship period. We found that increasing the proportion of female mentors tends to decrease the impact of protégés, and also found that the gain of female mentors decreases when mentoring female instead of male protégés.

While it has been shown that having female mentors increases the likelihood of female protégés staying in academia [9] and provides them with better career outcomes [35], such studies often compare protégés that have a female mentor to those who do not have a mentor at all, rather than to those who have a male mentor. In this paper, we addressed this limitation in the context of informal mentorship, and found that female protégés who remain in academia reap more benefits when mentored by males rather than females, while the benefits of mentorship accruing to senior female mentors are also higher when working with otherwise similar male protégés. These findings suggest that current diversity policies promoting female-female mentorships, as well-intended as they may be, could hinder the careers of women who remain in academia in unexpected ways. Female scientists, in fact, may benefit from encouraging opposite-gender mentorships instead throughout their careers. Policy makers should thus revisit first and second order consequences of diversity policies while focusing not only on retaining women in science, but also on maximizing their long term scientific impact. In addition to these perspectives, policies should also consider the benefits accrued from gender diversity within scientific teams, as well as the quality of their overall scientific output [38]. More broadly, the goal of gender equity in science, regardless of the objective targeted by these policies, cannot, and should not be shouldered by senior female scientists alone, rather, it should be embraced by the scientific community as a whole.

Methods

The data used for this study was obtained in October 2015 from the Microsoft Academic Graph (MAG) dataset [36]. This data set includes records of scientific publications specifying the date of the publication, the authors’ names and affiliations, the publication venue, and the keywords. It also contains a citation network in which every node represents a paper and every directed edge represents a citation. While the number of citations of any given paper is not provided explicitly, it can easily be calculated from the citation network in any given year. Additionally, every keyword in a given paper is positioned in a field-of-study hierarchy, the highest level of which is comprised of 19 scientific disciplines.

Using the information provided in the MAG dataset, we derive two key measures: the main discipline of scientists and their impact. In particular, we classify scientists into disciplines using the method proposed by [18]. This method classifies any given scientist, $s_{i}$, using the keywords that are specified in the publications of $s_{i}$, which are themselves classified into disciplines by MAG. On the other hand, the impact of each scientist in any given year is derived from the citation network provided by MAG. In addition to the scientists’ discipline and impact, we derive additional measures such as the scientists’ gender, which is determined using “genderize.io” [39], and the rank of each university, which is determined based on the 2018 Academic Ranking of World Universities, also known as the Shanghai ranking [40].

Whenever a junior scientist (with academic age $\leq 7$) publishes a paper with a senior scientist (academic age $>7$), we consider the former to be a protégé, and the latter to be a mentor. We consider the start of the mentorship period to be the year of the first publication of the protégé, and consider the end of the mentorship period to be the year when the protégé becomes a senior scientist. We analyze every mentor-protégé dyad that satisfies all of the following conditions: (i) the protégé has at least one publication during their senior years without a mentor; (ii) the affiliation of the protégé is in the United States throughout their mentorship years; (iii) the main discipline of the mentor is the same as that of the protégé; (iv) the mentor and the protégé share an affiliation on at least one publication; and (v) the protégés do not have a gap of 5-years or more in their publication history. As a consequence, our analysis excludes all scientists: (i) who never published any papers without their mentors post mentorship, as we cannot analyze their scientific impact in their senior years independent of their mentors; (ii) who only had solo-authored papers or collaborations with their junior peers or with seniors from other universities, as we cannot clearly establish who their mentors were; (iii) who had a gap longer than 5-years without any publications; and (iv) who only collaborated with senior scientists outside of their main discipline.

As our use sample we consider the ten disciplines in MAG that have the largest number of mentor-protégé pairs, namely Biology, Chemistry, Computer Science, Economics, Engineering, Geology, Mathematics, Medicine, Physics, and Psychology (see Supplementary Table S1). From these we drop Physics since the mean number of authors per paper in this discipline is far larger than that of any other discipline (236, compared to 13 or below; see Supplementary Table S2). As such, keeping Physics would render our approach of identifying mentor-protégé relationships quixotic and impractical. Furthermore, we dropped all mentor-protégé pairs where the gender of the protégé could not be established with at least 95% certainty. Later on, when analyzing the gender of mentors, we also drop any mentors whose gender could not be established with at least 95% certainty.

A total of 140 different Coarsened Exact Matchings (CEMs) were used to produce the results depicted in Fig. 1, Fig. 2, Fig. S1, Fig. S2, and Fig. S3. Additionally, a total of 32 different matchings were used to produce the results depicted in Fig. 3. More details about the confounding factors used therein, as well as the binning decisions, can all be found in the Supplementary Note 2.