Should we replace the traditional ways of evaluating scientific quality with indices of mentorship?

July 8, 2021 • 12:30 pm

A new paper in PLoS Biology (click on screenshot below) calls for a thorough revamping of the way scientists are evaluated for the quality of their work, replacing traditional methods of assessment (research productivity and quality) with evaluations of “mentorship”. The reason the authors want to dismantle the traditional “meritocratic” methods of evaluation (based, they say, but erroneously, on citation rates) is that these methods are biased against women and minorities. But their evidence for that is almost nonexistent, and, in the end, what they are doing is replacing the traditional empirical purpose of science—to understand the universe—with a social purpose: to increase social justice.

I want to emphasize at the outset that insofar as science is racist, with the racism built into the system, that needs to be changed. The authors feel that science is deeply racist, with the result that women and minorities don’t get cited, don’t get grants, don’t get tenure, and in general achieve less than white people or men. If that is due to current practices in science, or current biases of scientists, then this must change. But the evidence that structural racism is pervasive in science today is nonexistent. (Some scientists, of course, are racists, but the system itself, I claim, is now set up to favor women and minorities, not hold them down.)

Further, as I’ve said before, we do need a form of affirmative action in education and in science. It simply won’t do if the present system somehow results in a glaring deficit of women and people of color in science. I think more equitable representation (though not necessarily proportional representation) is a moral imperative, if for no other reason than to provide a form of reparations for groups who were held back years ago and haven’t yet caught up. What I do claim is that, at present, science is not nearly as racist as the authors represent.

In fact, if you’re involved in American academic science, you know that departments are scrambling hard to get minority and women faculty and graduate students. The reason we have trouble getting minorities, however, is the “pipeline problem”, based on a system of oppression and cultural differences that traces back centuries, and must be rectified not by changing the criteria for advancing in science, or sniffing out a “systemic racism” in science doesn’t exist, but by allowing everyone to have an equal opportunity to become a scientist. And that involves big societal changes that afford equal opportunity from birth.

I’ve said all this before, so let me present the authors’ abstract, which pretty much tells their tale:

Success and impact metrics in science are based on a system that perpetuates sexist and racist “rewards” by prioritizing citations and impact factors. These metrics are flawed and biased against already marginalized groups and fail to accurately capture the breadth of individuals’ meaningful scientific impacts. We advocate shifting this outdated value system to advance science through principles of justice, equity, diversity, and inclusion. We outline pathways for a paradigm shift in scientific values based on multidimensional mentorship and promoting mentee well-being. These actions will require collective efforts supported by academic leaders and administrators to drive essential systemic change.

In short, the authors want to replace citations (who cites your papers in their papers, and how often) and “impact factors” (the quality of the journals in which you publish) with “multidimensional” and “holistic” mentorship—a mentorship not designed to produce scientists who find out things, but who are healthy, happy, mentally stable, dedicated to equity, and representing all ethnic groups. This is a proposal to make science into a vehicle for social engineering.

Here’s a figure in which the authors purport to show the problem.

Note that their characterization of how science is done now is said to focus exclusively on “citations” as a measure of one’s impact on the field (encompassed by the “H index“, which didn’t exist for most of my career), while the “inclusive view” focuses on basically everything.

Much of the PLOS paper focuses on the inequalities said to be faced by both women and minorities: fewer citations, fewer awards, less grant funding, greater difficulties in publication, and so on. If one evaluates people based on these factors, especially citations, one is said to be exercising biases that hold down women and minority representations.

But I know of no academic vetting process that just counts citations and ignores mentorship as a way of evaluating quality. What is important is not just impact measured by citations, but by the quality of one’s work, assessed in many different ways. The most important is reading your papers, seeing what you’ve found out, and learning whether other scientists express interest in your work. Have you pushed the field forward with interesting findings? Have you produced some accomplished students (which, by the way, is mentorship)? In the promotion and tenure committees I’ve been on, citation counts have not ever been mentioned. Rather, a candidate’s papers are read and discussed, letters solicited from people in the field are considered, production of graduate students is noted, teaching is assessed (not so heavily at a research school), and—in some places, but not Chicago—outside research support is assessed. Citation numbers are only a small part of this process, and at Chicago weren’t even considered when I was on the promotion and tenure committee in biology.

Are women and minorities really held down by the sexism and structural racism of academic science, though? The evidence is thin.  The authors cite older papers in support of this thesis, but completely ignore newer papers  showing that virtually every metric of women’s advancement in science is now on par with men, that there is no perceptible bias against either women or minorities in assessing grant proposals (nobody ever cites this important paper because it doesn’t meet what people expected to find), that the lower rate of funding by black scientists isn’t due to biases against them, but to differential choice of fields having different amounts of funding, and a survey of faculty and student hiring preferences showed that minority candidates and women are preferred above white candidates and males, especially by faculty. None of these papers are cited by Davies et al.

Removing not just citations but other ways of evaluating scientific productivity—all aimed at answering the question, “Has this person pushed our knowledge forward?”—is a surefire way to erode the quality of science. As I said, our main aim is to find out stuff, not act as a vehicle for social justice, though we should of course behave towards our students and colleagues in an unbiased fashion.

What about replacing the traditional criteria with measures of mentorship? This itself involved problems, because, at least for academic mentorship, exposure to science means exposure to research. One of the signs of a good mentor, like Dick Lewontin, is that they produce lots of students who themselves produce lots of research, so their reach is extended.  You could use grand-citations to do this, but traditionally a “holistic” metric is used: how much knowledge has come, directly or indirectly, from this person?

But what if your goal is to produce teachers or workers in industry rather than researchers?  First, having research experience always helps you get a job in industry. As for teaching, I would argue that exposure to research, even if it isn’t published, is an essential part of producing someone who’s a good teacher of science. Good teachers understand how research is done, and you need to learn to do that by doing it, not by reading about it. In the end, research should always be part of any scientific training. And if you’re at a teaching school, or teaching science in secondary school, citations and research are of minor or no importance.

But Davies et al. aren’t interested in this kind of mentorship. They want a “holistic” mentorship in which research is downplayed in favor of producing students who conform to social-justice expectations. These students are mentally healthy as well as having “bystander intervention training” and “anti-bullying and antiracist mentoring and teaching practices”. One’s mentorship must “promote justice, equity, diversity, and inclusion in science.”  Other practices of good mentors include these:

To ensure that training opportunities become valued by participants, institutions may consider implementing mandatory participation by requiring training for career advancement or as prerequisites for recruiting mentees. However, training programs should be mindfully designed to engage those who may complete training for inauthentic reasons. [JAC: What???] Discussions of topics covered in training should become standard practice at regular events including faculty meetings and retreats and graduate student association meetings. Undergraduate programs can include discussions of unconscious bias and how such biases influence classroom dynamics.

And we must become experts in mental health as well:

While good mentorship can foster a sense of belonging in science for the mentee, relationships of many mentees from marginalized groups with their mentors—who are often from the majority group—are not always positive, leading to health issues, such as insomnia and anxiety, and lower retention of these groups in science (reviewed in [93,104]). In order to effectively mentor, all mentors—particularly those who are not familiar with the experiences and perspectives of systemically marginalized scholars—should engage with cultures, communities, and perspectives that differ from their own, connect with communities that are working toward creating justice, equity, diversity, and inclusion, and support institutional change already underway. In addition, increasing representation from marginalized communities throughout institutional hierarchies provides greater opportunities for mentees to find mentors with which to build meaningful relationships.

Of particular concern is the recently highlighted decline in mental health of many academics and a growing crisis at the graduate level. Graduate students are at least twice as likely to experience mental health challenges, such as anxiety and depression, compared to the general population with equivalent education. This trend is even more striking for women of color in STEMM, who are facing systemic sexism and racism, along with daily microaggressions and safety concerns. Sexual minorities and LGBTQ+-identifying people are also subject to discrimination that adversely affects their well-being, mental health and, ultimately, retention in STEMM fields. Laboratory work, field work, and simple existence in the academy can often place marginalized groups, including those with disabilities, at risk of injury, harassment, bullying, and assault). To combat these challenges, specific strategies for safety and well-being must be supported at the research group, departmental, institutional, and funding organization levels.

Multidimensional mentorship clearly requires expertise in psychotherapy.

While good mentors are sensitive to their student’s psychological needs, there are other groups within universities designed deal with these issues. And emphasis on all of them dilutes the very purpose why one does science, turning it into a vehicle for promoting “social justice” in the community. But that’s exactly what the authors want.

I am drawing to a close, for I’ve seen paper after paper like this in the last few weeks, all suggesting that we ditch traditional ways of assessing scientific merit, because those ways give a disadvantage to minorities and diminish social justice.

I would suggest, though, that there are better vehicles than science for promoting social justice, and that we shouldn’t turn our field into an ideological juggernaut. Of course we must provide equal opportunities for all, but I think academic science is doing a pretty good job at that. There will surely be dissenters touting their own lived experience of oppression, but the data we have now suggests that science is doing a pretty good job at promoting equal opportunity. And I have not yet heard a way to improve the mission of science—to find out truth—than the ways we’re doing it now. We’re casting wider nets for talent, and fighting hard to eliminate any biases that we can find.  All I can say is that I disagree with the idea of replacing the criteria currently used to evaluate someone as a scientist with criteria mainly concerned with mentoring people in a social-justice-y way.

19 thoughts on “Should we replace the traditional ways of evaluating scientific quality with indices of mentorship?

  1. Last night I read this article in Quillette about a college teacher’s teaching experience….including at a community college. The article is written pseudonymously. It’s not a short article, but was wondering if any of you could share thoughts on it….especially if you have been academic teachers at any level.

    “The job I had been trained to do—help students work with the nuts and bolts of language as writers and readers, as well as help them (in the best of worlds) appreciate the power and beauty of written English—became more and more difficult. Some students considered questioning and criticism racist—and the texts we read and wrote about white. Such thinking expanded, in time, to embrace a variety of identities.”

  2. I would suggest, though, that there are better vehicles than science for promoting social justice, and that we shouldn’t turn our field into an ideological juggernaut.

    Quite so. There may be a moral imperative for ensuring equitable representation… but it’s a Goldilocks problem. You don’t want too little, you want something that is just right. And too much would be a bad thing.

  3. Yet another effort to guarantee mediocrity across the board—a kind of analogue in academia of thermodynamic ‘heat death’. My first instinct, when I see something like this, is to evaluate the author(s) of the proposal according to the criteria that they’re disparaging. Basically, have any of these people ever done anything seriously notable in biology? In the old days, before the tidal wave of wokebabble terminology, one’s reaction would very likely have been, this sounds an awful lot like sour grapes.

    I can just imagine Marie Curie’s, Emmy Noether’s, or Barbara McClintock’s reaction to something like this…

  4. In my experience, citing a paper is a colorblind and gender-blind process. No one looks to see whether the author(s) are women or black when deciding whether to cite a paper.

    1. Yes. I’d guess that the main basis for citing other people’s work in a paper, by a very long way, is the degree of support it gives to the claim or proposal advanced in that paper.

  5. I know that what I am going to say is strictly speaking unrelated, but I also think it’s much more urgent.

    I don’t know about the merits of the topic in discussion specifically, and I actually don’t care much about the goal of social justice in this instance. I care more about truth. And as far as truth goes, I have empirical reasons to think that for instance the current system of anonymous peer-review is either useless or downright harmful if you want a correct application of the scientific method, and that the entire socioeconomic ecosystem of scientific journals has become highly susceptible to monetary and political influences of all sorts (by autocratic states too for instance).

    I am really scared for the future of science and truth. I think all honest scientists should be scared.

  6. Leadership in science in the West is doomed, but Russia, China, Japan, and India will probably fill the void. Perhaps in the future, the students who want to learn STEM will strive to get into elite universities in those countries.

  7. ‘holistic’ – that magic word which lets you select and rearrange the data according to your prejudices.

  8. For a paper calling for greater inclusivity and diversity in science, it is striking that ALL of the authors are female.

  9. The authors of this paper have unaccountably neglected the most glaring example of nonDiversity, inEquity and unInclusion in modern science: its drastic exclusion of scientific reports in Russian, particularly those published in the USSR in the 1930s, 40s, and 50s. As a result of this systemic Latin alphabet supremacism and narrow outlook, Western scientists rarely cite the Russian literature that includes such classics as I. I. Prezent’s “Class Struggle on the Natural Science Front” (1932); Trofim Lysenko’s work on the change of plant species by manipulating the temperature; G.M. Boshyan’s announcement of the direct transformation of antibiotics into viruses; and the discovery of the spontaneous generation of cells from non-living crystals by Olga Lepeshinskaya (Stalin Prize, 1950).

  10. I know several of those authors, and Jerry probably knows one of them (Brooke Weigel from the Committee on Evolutionary Biology at the University of Chicago). They’re all afaik sincere and earnest in their efforts to make their communities into better places.

    They are also extremely accomplished researchers and scholars: at least one is a department chair; all of them publish important, highly visible research in marine ecology and conservation (at least I think that’s the main area of research the authors all share in common).

    But the authors undermine their own arguments about over-emphasis on citation metrics in career advancement. That paper could have been posted as a preprint on bioRxiv, but instead the authors paid $4000 to publish it in a gold open access PLoS journal with a high impact factor (>7.0). Why? One guesses because the highly selective nature of the journal and its high IF increase the prestige associated with publishing the paper there. bioRxiv is a dump by comparison.

    Also the first author prominently displays her own bibliometrics achievements (total citations 1109, H index 16, i10 index 22) on her Boston University CV. There may be good reasons for women to play the impact factor/H index game even while arguing the game is sexist, but it still seems disingenuous to boast about doing well in such an apparently deplorable system.

  11. I rather think that science and university education should stop being a take-no-prisoner world that squeezes and then discards people, and let those who satisfy certain standards live and work (I don’t begrudge giving top positions and most of grant money to the most productive). To me, there is no valid reason for scientists to work twice as long hours as most other employees in the same country (often for miserable pay) and always to be forced to feel that they are not performing well enough. My university values citations highly. For this reason, I intend to take a turn from fundamental studies on gamete cell biology, which I love but are not cited enough, to something more medically oriented in order to harvest more citations. I have already reached the top of my career, so I wouldn’t do it if it was about me, but my younger colleagues will need citations to get tenures.

    About women – the way science is organized is inherently set against reproducing females. First, the neverending years of study (Bachelor, Master, PhD, postdoc) dangerously approach the full duration of a woman’s reproductive life. Second, if she manages to reproduce after all, how can she maintain the insane productivity that is required and at the same time parent her child(ren)? But I hate to use female experience to argue that scientists of both sexes need to be treated like human beings. Male scientists also should stop being students by age 30, and they need time to parent their children if they have any, or just rest or go fishing or whatever.

  12. I feel that the first-past-the-post (FPTP) electoral system just barely qualifies as democratic rule within the democracy spectrum, though it seems to serve corporate interests well. I believe it is basically why powerful money interests generally resist attempts at changing from FPTP to proportional representation electoral systems of governance, the latter which dilutes corporate lobbyist influence.

    American and Canadian governances typically maintain thinly veiled yet strong ties to large corporations, as though elected heads are meant to represent big money interests over those of the working citizenry and poor. Accordingly, major political decisions will normally foremost reflect what is in big business’s best interests. And don’t expect to hear this fact readily reported by the mainstream news-media, which is concentratedly corporate owned.

    “Now you’re not naive enough to think we’re living in a democracy, are you, Buddy? It’s the free market, and you’re part of it.”
    —the morbidly greedy bank-financier Gordon Gekko to his young stockbroker protégé Bud Fox (Wall Street, 1987)

  13. “In the end, research should always be part of any scientific training. And if you’re at a teaching school, or teaching science in secondary school, citations and research are of minor or no importance.”

    Due to increasingly common privatized research for corporate profit aims, even ‘scientific fact’, to a concerning degree, is for sale. Research results, however flawed, can and are known to be publicly amplified if they favor the corporate product, and accurate research results can be suppressed if they are unfavorable to business interests, even when involving human health. Health Canada was established to act in Canadian consumers’ best interests yet is susceptible to corporate lobbyist manipulation.

    For one thing, it allowed novelty-flavored vaping products to be fully marketed — even on corner stores’ candy counters — without conclusive independent scientific proof that the product, as claimed by the tobacco industry, would not seriously harm consumers but rather help nicotine addicts wean themselves off of the more carcinogenic cigarette means of nicotine deliverance. A few years before that, Health Canada had sat on its own research results that indicated seatbelts would save lives and reduce injury; it wanted even more proof of safety through seatbelts before ordering big bus manufacturers to install them in every bus.

    To me, those examples smell of science-be-damned lobbyist manipulation — something that should not prevail in a government body established primarily, if not solely, to protect consumers’ safety and health rather than big businesses’ monetary concerns.

Leave a Reply