Should we replace the traditional ways of evaluating scientific quality with indices of mentorship?

A new paper in PLoS Biology (click on screenshot below) calls for a thorough revamping of the way scientists are evaluated for the quality of their work, replacing traditional methods of assessment (research productivity and quality) with evaluations of “mentorship”. The reason the authors want to dismantle the traditional “meritocratic” methods of evaluation (based, they say, but erroneously, on citation rates) is that these methods are biased against women and minorities. But their evidence for that is almost nonexistent, and, in the end, what they are doing is replacing the traditional empirical purpose of science—to understand the universe—with a social purpose: to increase social justice.

I want to emphasize at the outset that insofar as science is racist, with the racism built into the system, that needs to be changed. The authors feel that science is deeply racist, with the result that women and minorities don’t get cited, don’t get grants, don’t get tenure, and in general achieve less than white people or men. If that is due to current practices in science, or current biases of scientists, then this must change. But the evidence that structural racism is pervasive in science today is nonexistent. (Some scientists, of course, are racists, but the system itself, I claim, is now set up to favor women and minorities, not hold them down.)

Further, as I’ve said before, we do need a form of affirmative action in education and in science. It simply won’t do if the present system somehow results in a glaring deficit of women and people of color in science. I think more equitable representation (though not necessarily proportional representation) is a moral imperative, if for no other reason than to provide a form of reparations for groups who were held back years ago and haven’t yet caught up. What I do claim is that, at present, science is not nearly as racist as the authors represent.

In fact, if you’re involved in American academic science, you know that departments are scrambling hard to get minority and women faculty and graduate students. The reason we have trouble getting minorities, however, is the “pipeline problem”, based on a system of oppression and cultural differences that traces back centuries, and must be rectified not by changing the criteria for advancing in science, or sniffing out a “systemic racism” in science doesn’t exist, but by allowing everyone to have an equal opportunity to become a scientist. And that involves big societal changes that afford equal opportunity from birth.

I’ve said all this before, so let me present the authors’ abstract, which pretty much tells their tale:

Success and impact metrics in science are based on a system that perpetuates sexist and racist “rewards” by prioritizing citations and impact factors. These metrics are flawed and biased against already marginalized groups and fail to accurately capture the breadth of individuals’ meaningful scientific impacts. We advocate shifting this outdated value system to advance science through principles of justice, equity, diversity, and inclusion. We outline pathways for a paradigm shift in scientific values based on multidimensional mentorship and promoting mentee well-being. These actions will require collective efforts supported by academic leaders and administrators to drive essential systemic change.

In short, the authors want to replace citations (who cites your papers in their papers, and how often) and “impact factors” (the quality of the journals in which you publish) with “multidimensional” and “holistic” mentorship—a mentorship not designed to produce scientists who find out things, but who are healthy, happy, mentally stable, dedicated to equity, and representing all ethnic groups. This is a proposal to make science into a vehicle for social engineering.

Here’s a figure in which the authors purport to show the problem.

Note that their characterization of how science is done now is said to focus exclusively on “citations” as a measure of one’s impact on the field (encompassed by the “H index“, which didn’t exist for most of my career), while the “inclusive view” focuses on basically everything.

Much of the PLOS paper focuses on the inequalities said to be faced by both women and minorities: fewer citations, fewer awards, less grant funding, greater difficulties in publication, and so on. If one evaluates people based on these factors, especially citations, one is said to be exercising biases that hold down women and minority representations.

But I know of no academic vetting process that just counts citations and ignores mentorship as a way of evaluating quality. What is important is not just impact measured by citations, but by the quality of one’s work, assessed in many different ways. The most important is reading your papers, seeing what you’ve found out, and learning whether other scientists express interest in your work. Have you pushed the field forward with interesting findings? Have you produced some accomplished students (which, by the way, is mentorship)? In the promotion and tenure committees I’ve been on, citation counts have not ever been mentioned. Rather, a candidate’s papers are read and discussed, letters solicited from people in the field are considered, production of graduate students is noted, teaching is assessed (not so heavily at a research school), and—in some places, but not Chicago—outside research support is assessed. Citation numbers are only a small part of this process, and at Chicago weren’t even considered when I was on the promotion and tenure committee in biology.

Are women and minorities really held down by the sexism and structural racism of academic science, though? The evidence is thin.  The authors cite older papers in support of this thesis, but completely ignore newer papers  showing that virtually every metric of women’s advancement in science is now on par with men, that there is no perceptible bias against either women or minorities in assessing grant proposals (nobody ever cites this important paper because it doesn’t meet what people expected to find), that the lower rate of funding by black scientists isn’t due to biases against them, but to differential choice of fields having different amounts of funding, and a survey of faculty and student hiring preferences showed that minority candidates and women are preferred above white candidates and males, especially by faculty. None of these papers are cited by Davies et al.

Removing not just citations but other ways of evaluating scientific productivity—all aimed at answering the question, “Has this person pushed our knowledge forward?”—is a surefire way to erode the quality of science. As I said, our main aim is to find out stuff, not act as a vehicle for social justice, though we should of course behave towards our students and colleagues in an unbiased fashion.

What about replacing the traditional criteria with measures of mentorship? This itself involved problems, because, at least for academic mentorship, exposure to science means exposure to research. One of the signs of a good mentor, like Dick Lewontin, is that they produce lots of students who themselves produce lots of research, so their reach is extended.  You could use grand-citations to do this, but traditionally a “holistic” metric is used: how much knowledge has come, directly or indirectly, from this person?

But what if your goal is to produce teachers or workers in industry rather than researchers?  First, having research experience always helps you get a job in industry. As for teaching, I would argue that exposure to research, even if it isn’t published, is an essential part of producing someone who’s a good teacher of science. Good teachers understand how research is done, and you need to learn to do that by doing it, not by reading about it. In the end, research should always be part of any scientific training. And if you’re at a teaching school, or teaching science in secondary school, citations and research are of minor or no importance.

But Davies et al. aren’t interested in this kind of mentorship. They want a “holistic” mentorship in which research is downplayed in favor of producing students who conform to social-justice expectations. These students are mentally healthy as well as having “bystander intervention training” and “anti-bullying and antiracist mentoring and teaching practices”. One’s mentorship must “promote justice, equity, diversity, and inclusion in science.”  Other practices of good mentors include these:

To ensure that training opportunities become valued by participants, institutions may consider implementing mandatory participation by requiring training for career advancement or as prerequisites for recruiting mentees. However, training programs should be mindfully designed to engage those who may complete training for inauthentic reasons. [JAC: What???] Discussions of topics covered in training should become standard practice at regular events including faculty meetings and retreats and graduate student association meetings. Undergraduate programs can include discussions of unconscious bias and how such biases influence classroom dynamics.

And we must become experts in mental health as well:

While good mentorship can foster a sense of belonging in science for the mentee, relationships of many mentees from marginalized groups with their mentors—who are often from the majority group—are not always positive, leading to health issues, such as insomnia and anxiety, and lower retention of these groups in science (reviewed in [93,104]). In order to effectively mentor, all mentors—particularly those who are not familiar with the experiences and perspectives of systemically marginalized scholars—should engage with cultures, communities, and perspectives that differ from their own, connect with communities that are working toward creating justice, equity, diversity, and inclusion, and support institutional change already underway. In addition, increasing representation from marginalized communities throughout institutional hierarchies provides greater opportunities for mentees to find mentors with which to build meaningful relationships.

Of particular concern is the recently highlighted decline in mental health of many academics and a growing crisis at the graduate level. Graduate students are at least twice as likely to experience mental health challenges, such as anxiety and depression, compared to the general population with equivalent education. This trend is even more striking for women of color in STEMM, who are facing systemic sexism and racism, along with daily microaggressions and safety concerns. Sexual minorities and LGBTQ+-identifying people are also subject to discrimination that adversely affects their well-being, mental health and, ultimately, retention in STEMM fields. Laboratory work, field work, and simple existence in the academy can often place marginalized groups, including those with disabilities, at risk of injury, harassment, bullying, and assault). To combat these challenges, specific strategies for safety and well-being must be supported at the research group, departmental, institutional, and funding organization levels.

Multidimensional mentorship clearly requires expertise in psychotherapy.

While good mentors are sensitive to their student’s psychological needs, there are other groups within universities designed deal with these issues. And emphasis on all of them dilutes the very purpose why one does science, turning it into a vehicle for promoting “social justice” in the community. But that’s exactly what the authors want.

I am drawing to a close, for I’ve seen paper after paper like this in the last few weeks, all suggesting that we ditch traditional ways of assessing scientific merit, because those ways give a disadvantage to minorities and diminish social justice.

I would suggest, though, that there are better vehicles than science for promoting social justice, and that we shouldn’t turn our field into an ideological juggernaut. Of course we must provide equal opportunities for all, but I think academic science is doing a pretty good job at that. There will surely be dissenters touting their own lived experience of oppression, but the data we have now suggests that science is doing a pretty good job at promoting equal opportunity. And I have not yet heard a way to improve the mission of science—to find out truth—than the ways we’re doing it now. We’re casting wider nets for talent, and fighting hard to eliminate any biases that we can find.  All I can say is that I disagree with the idea of replacing the criteria currently used to evaluate someone as a scientist with criteria mainly concerned with mentoring people in a social-justice-y way.