The New York Times recently had a piece by their new and woke science reporter, Katherine J. Wu, which is basically an indictment of science journals for not keeping track of the “diversity” of authors and reviewers of the papers they publish or reject. The implicit message is that science journals are racist, discriminating against papers by minoritized authors.
Click on the screenshot to read the article:
Wu’s implicit assumption is twofold. First, that a paucity of diversity—which of course means ethnic diversity, but minus Asians since they are surely overrepresented among authors—reflects racism on the part of scientific journals and reviewers. There is no consideration of whether a lack of diversity may represent simply a paucity of minority authors and reviewers. That itself may reflect racism, past or present, that narrows the opportunities of would-be scientists, but the article implies that it’s racism acting on Ph.D. authors trying to submit papers.
The second assumption is that more ethnic diversity in journals means better science. Well, that’s true in the sense that the more people who get the opportunities to become scientists, the higher the average quality of the science that is published. But I’m not at all convinced that members of any group, be they groups involving genders, religions, incomes, or ethnicity, have a special “point of view” based on their group identity that makes them do science differently. Science is science, and I don’t feel that Hispanics, say, have a different “way of knowing.” (There may be one exception here, that I’ve mentioned before: I think women scientists are responsible for shifting the focus in sexual selection from male traits alone to female preferences as well. But many men were also involved in this shift). In the end, the best science comes from giving everybody equal opportunities, not practicing remediation based on race at the publication level.
But the question is whether journals should be publishing more papers by members of minority groups. That is, is there a bias against, say, black or Hispanic authors that needs to be rectified by that form of “affirmative action” on the publication level—taking steps to accept more papers by minority authors?
It’s my opinion that the answer is “no”. This presumes that a paucity of papers by such authors is prima facie evidence for bias, when it may reflect only a paucity of minority-group members in the field, or of minority scientists submitting papers, or submitting fewer papers,—rather than reviewers deliberately discriminating against papers by minority authors.
It may be worth investigating this issue, but I consider it hardly worthwhile for two reasons. First, figuring out whether a paucity of papers from minority group members is due to racism at the reviewing level is very hard to do, though not impossible (see below). More important, it’s certainly true that the disparity between the proportion of minority-group members in the population and the number of papers published by members of that group is due largely not to racism but to an underrepresentation of Blacks and Hispanics in science. Figuring out why that disparity exists is the best way to achieve more proportionality in science, if that is your goal. And that’s really where our efforts should be going.
Here are the data given in the NYT piece from two scientific organizations showing disparities between population proportions and publication/reviewer proportions. The article makes the point that most journals, though, do not keep records of the ethnicity of authors and reviewers. (To clarify for non-scientists, when scientists submit papers to a journal, those papers are sent to several reviewers—usually two or three—who are experts in the area of research. Based on the reviewers’ assessment of the paper, the editor then decides whether or not to publish it. If the decision is “yes,” there is often some revision of the paper required, either in the discussion or the scientific analysis.)
I’m going to discuss authors here, not reviewers, because it is the quality of authors‘ work that, by and large, constructs the quality of the journal. How do we know if a journal is discriminating against minority authors? You can’t simply use a difference between the proportion of people in the field, or the proportion of people submitting papers on the one hand, and the proportion of papers published on the other, as a criterion for bias. That’s because members of different groups may submit papers less often, or of lesser quality, and that this would lead to differential representation that would not reflect racism. Bias must be proven, not assumed.
There are two ways to solve this problem. The first is the equivalent of doing “blind” auditions for orchestras—auditions in which those seeking an orchestral chair perform behind a curtain. That “blind” system removes all bias against sex or race. To do this with a paper, you simply remove the names of the authors, their institution, and the acknowledgments from the manuscript, so the reviewers don’t know who wrote the paper. (There are, of course, ways to guess, like if an author cites herself repeatedly, but in many cases this will indeed lead to quality appraisal ignorant of the author’s race or gender.)
I hit upon that system in the late 1970s when I was a postdoc, full of piss and vinegar and concerned that papers were getting preferentially published not because of race, but because of reputation. My idea was that famous people had an easier time publishing their papers than small fish (like me!). I wrote letters—real letters—to the editors of about 30 journals in my field, proposing that manuscripts be reviewed blind this way. I got only one response, and that was from an editor who said that he preferred knowing the authors, because famous authors were more likely to submit better papers! That may be true on average, but it’s not the best way to ensure the quality of papers in a journal! In fact, famous authors may get by more easily with shoddier work because of their reputations.
At any rate, some journals have now wisely decided to adopt the blind-author technique, and more power to them! It seems to me a step in the right direction to eliminate animus not just against groups of people, but against your scientific “enemies” or in favor of your scientific “friends”. (Believe me, this kind of bias is rife in science.) While you can get around this system by guessing, I think it does help ensure objective reviewing and thus higher-quality papers. (I should add that the NYT music critic opposed blind auditions because he said that while it increased the proportion of women in orchestras, it didn’t eliminate racial inequities; his view was clearly that equity trumped orchestral quality.)
The other way would be to do an experiment submitting identical sets of manuscripts with fake names that give clues to the gender or ethnicity of the authors. If manuscripts with women or minority authors are rejected more often than the same manuscripts with “white” or “male” names, that surely indicates bias. This was what was done in a laborious study of grant reviewing, using made-up “black”, “white”, “male” and “female” names on identical proposals. The study showed no evidence of racial or gender bias in grant evaluation. Needless to say, you don’t hear much about this study, even though it was a good one, as the results went against people’s certainty that there must be sex and gender bias in reviewing.
That experiment could be done with paper reviewing too, and really must be done before you can start making implicit accusations of bias. But I favor the blind-reviewing technique. You don’t have to do any experiments to see if that one makes things more equitable because, by eliminating a source of bias from the outset, it almost has to. It is my feeling that a “fake name” study wouldn’t show evidence of bias in pubication, but that’s my feeling alone. Better just to practice blind reviewing rather than speculate or do experiments.
In the end, my feeling is that affirmative action should not be applied to reviewing papers by people who already have doctorates, and, while I believe in affirmative action, I think it has to stop at some point in the hierarchy. My point comes after faculty hired hiring. I think it’s okay and useful to take race and gender into account when hiring junior faculty, as well as in college and grad-school admissions, but that’s where it stops. Ethnicity and gender should not be a consideration in getting tenure, full professorships, or in getting papers published—areas where merit alone should be the only criterion. Again, this is my view, and others may disagree.
Some of those who disagree think the whole system of a scientific meritocracy is flawed—that there isn’t even a scientific meritocracy. The NYT article says that:
Publishing papers in top-tier journals is crucial scholastic currency. But the process is deeply insular, often hinging on personal connections between journal editors and the researchers from whom they solicit and receive manuscripts.
“Science is publicized as a meritocracy: a larger, data-driven enterprise in which the best work and the best people float to the top,” Dr. Extavour said. In truth, she added, universal, objective standards are lacking, and “the access that authors have to editors is variable.”
To democratize this process, editors and reviewers need to level the playing field, in part by reflecting the diversity that journals claim they seek, Dr. Kamath said. “People think this is a cosmetic or surface issue,” she said. “But in reality, the very nature of your scholarship would change if you took diversity, equity and inclusion seriously.”
This whole section is to imply that there is little correlation between the merit of a paper and the chance of its being published. I think that’s a foolish conclusion, with the caaveats that Wu gives meant to imply a weak correlation at best. This is not my experience in reviewing papers or assessing published papers. Yes, sometimes a terrible paper gets published in a good journal, and a great paper gets rejected by a good journal, but there is surely a correlation between the quality of a paper and the chance that a. it will get published, and b. that it will get published in a prestigious journal.
No, to democratize the process, just do blind reviewing. That will go a ways toward eliminating bias. But even in the absence of that procedure, journals would be hard pressed to construct a system that would give preferential publication to papers by ethnic minorities. Regardless of what Katherine Wu thinks, science largely is a meritocracy, at least when it comes to publication, and I don’t think it would be good for science as a whole to bump papers up or down based on the race of their authors.
Agree completely about blind reviewing. Of course, I suppose we should now include some non-scientists and stupid people to review scientific papers.
I like blind reviews, although you don’t see a lot of them. Although I’m not sure whether they lead to harsher or softer final assessments. Overall a serious flaw is a serious flaw no matter who wrote the thing.
In small fields it’s pretty easy from the subject matter to work out which lab a manuscript comes from, even if you don’t know the order of the authors. And, as you note, everyone cites themselves at some point, even if only on background – heaven forbid we should actually publish full methods with each paper!
PS Is WordPress still playing up? I’m logged in there, but not automatically here for some reason.
Ambika Kamath may not have been the best choice of protagonist for the NYT story. Dr. Kamath was a central character in this sad story about whole-sale invention of data for fraudulent publications.
https://dynamicecology.wordpress.com/2020/01/31/friday-links-240/
Dr. Kamath was victimized by the guy who carried out that fraud, and she was not responsible for the fraudulent papers. But Dr. Kamath got a huge career boost from those fake papers on her VC — about one-quarter of her total published research output was papers from that data fraud, coauthored with the guy who faked all the data. In addition to being victimized by the fraudster, Dr. Kamath has been widely criticized as well for benefitting (perhaps more than any other of the fraudster’s collaborators and coauthors).
From that pov, Dr. Kamath seems like the wrong person to be commenting on the quality of scientific publishing or the process that leads to published articles.
I just finished reading this and some of the links. Holy smokes, what a mess.
@Edward yes it is a slow-rolling disaster. Some but not all of the fraudulent papers have been retracted, the fraudster has lawyered up and sent threatening cease-and-desist letters to all kinds of people (including his former collaborators and coathors and to journal editors where the fraudulent papers were published), and those other folks have all gone to ground waiting for the fraudster’s university to investigate and report.
Dr. Kamath and the other coauthors were blindsided by this and have handled the fallout admirably. But she really should not be criticizing the publication process, or how it disadvantages people of color, given how that fraudulent process benefited her in this case. Again no personal criticism of her, just an observation that she is not the best spokesperson on this issue.
That probably wouldn’t have worked in my field; it was ‘intimate’ enough that you could pretty much tell who the authors were simply by the type and timing of the research. “Oh, I see Bob finally submitted on that experiment he did last year.”
But perfect isn’t the enemy of good; we can blind, and that still helps out some.
I partially disagree. Meaning I agree that we should not do it if that means valuing diversity over technical qualification of reviewers, however I see no reason not to strive for diversity if you have a bevy of qualified reviewers to choose from. If Alice, Bob, and Charlie are all great reviewers for a paper, and you only need two, why not pick Alice as one of them?
I agree with you that lack of diversity amongst reviewers in some fields is probably directly related to how many minority researchers there are in a field. However I’d say there’s at least one other barrier: editors relying on authors for recommendations as to reviewers. While this might sometimes be necessary (editors are busy, it’s often a part time job, and nobody is familiar with every nook and cranny of a research field), it can perpetuates an in-crowd, old-boys-network sort of approach to journal reviews. Journals, if you’re going to charge high subscription rates, put some of that money into building up a file of acceptable reviewers so you don’t have to ask the authors who should grade their work.
So, TL:DR – 1. IMO journals can get more reviewer-diverse without sacrificing any review quality by doing a better job of researching reviewers. 2. Along with that, stop relying on authors to recommend reviewers. 3. As with PCC’s recommended blind process, neither of these fixes is perfect, but they are all helpful.
This is why I didn’t mention reviewers and concentrated on authors. Yes, it would be good to get a diversity of reviewers, if for no other reason than to get everyone involved in the process (I don’t think “minority” reviewers would treat manuscripts in a consistently different way from other reviewers). But remember that many minority faculty complain because they are few and extra work is put upon them because of their ethnicity.
Yes, you’re right, we don’t want to pile extra work on people already overburdened with ‘extra-curricular’ duties. Having said that, peer reviewing other papers is a reasonably “core” activity for a researcher that I’d probably recommend getting rid of a bunch of other extra-curricular activities before that one.
Also I don’t think there’s anything too wrong if the system is better at inviting a more diverse set of researchers to review papers. You can always say no. And if the same non-prestigious journal keeps calling, “unsubscribe” – i.e. tell them to take you off their list and stop calling.
I think my focus was more on getting the journals themselves to modify their behavior to consider a wider range of reviewers. ‘Diversity’ shouldn’t mean researchers must accept an invitation to review if, for perfectly mundane reasons, they otherwise wouldn’t.
In my field (astrophysics) journals don’t ask authors for recommendations of referees.
The journals do, as you say, build up a database of expertise, and if they need more expert people to help with that they can invite more people onto the editorial board (many people will do that for little financial reward, for the kudos and as a service to the field).
It helps that the journals are mostly run by the learned societies as a service to the community, not by publishers for profit.
That sounds much better.
The question then is, I guess, do you have the database track race and use it to try and increase diversity, or do you have it intentionally not track race as a means of being neutral/blind to possible discrimination?
I’m a journal editor-in-chief. I am constantly looking out for manuscripts by Black and indigenous authors or other people of color that I could possibly help get through the peer review process. There are very few.
I’m also a university professor who trains graduate students and postdocs. In that role, I’m also scouring the landscape looking for Black and indigenous recruits to join my lab group. Again there are few (but I’ve been lucky to attract some of them).
The problem with invoking equity goals that reflect the demography of our cities and communities is that Black and indigenous people go into my field of research at much lower rates.
It seems clear (at least to me) that this is about economics and class, and not so much about race: Black and indigenous people tend to be much poorer, and fewer of them can afford to go to graduate school and live like starving artists for five or six years while getting a PhD. That’s the first problem to fix imho.
In a previous life I reviewed articles for 2 journals (not really top journals, I must admit), but the authors’ names and institutions were always blanked out. I had no idea about sex, ethnicity or academic provenance of the authors.
I’m puzzled that that appears not to be current practice. Of course, as eric points out, in a very restricted field you can guess, but you can guess wrong, of course.
Blind reviewing should be mandatory.
I agree. It’s hard to imagine any convincing reason WHY people wouldn’t submit papers “anonymously” and have them reviewed by people who have no idea of names, backgrounds, etc., or anything but the contents of the paper.
Not only could we guess the authors, but as authors, we could often guess who the reviewers were by the tone and focus of their writing. “Only Bob uses that phrase” or “Oh lord, there goes Bob again, talking about his data analysis pet peeve.”
Still, I don’t want anyone to think I’m down on the idea of blinding. It’s a valuable contribution to nondiscrimination, even if not the silver bullet.
That was always my one concern with the process — the reviewers and authors (and editors) of most specialized journals would recognize a blinded submission. Something like that wouldn’t necessarily privilege the known quantity, but there’s potential there. (There’s also the possibility that the reviewers know the author and think they’re a jerk or a poor scholar)
For editors and reviewers it is often helpful to know whether a weak paper with poor analysis and bad writing comes from a famous old fart who is just phoning it in, or from a junior researcher who is inexperienced.
Based on long experience, senior famous people often write bad papers and expect them to sail through, and those guys (it’s usually guys) often don’t respond well to constructive criticism. In those cases, reviewers and editors do well to just reject the paper.
But new inexperienced researchers who submit weak papers often benefit greatly from constructive feedback by reviewers and editors (which is really the point of peer review). In those cases, reviewers and editors do well to send helpful feedback and invite a revised manuscript.
That process can’t unfold in the way I described if reviewers and editors don’t know who the authors are. And that process is very common, it’s not a rare niche, and it’s probably much more common than any racist or sexist scenario in which reviewers recognize the authors’ ethnicity or gender and reject the paper on that basis.
But just my experience – ymmv.
I think you make a very good point, however what we’re talking about here is weighting the benefits of blinding vs. the reviewers time and effort. IOW, reviewers could still write thoughtful feedback for the old fogie set in his ways. The blinding process doesn’t prevent that. The fact that they see it as a waste of their uncompensated time prevents it. Which speaks to a possible solution which combines the benefits of in-depth peer response with the benefits of blinding, doesn’t it? 🙂
@Eric Yes I agree wrt uncompensated time and effort. And sure in principle reviewers could write thoughtful feedback for any author who submits any manuscript. I agree there are better ways to do this, and I agree blinding helps in some ways and is a hindrance in others.
It is *hard* for editors to find other researchers who both are knowledgeable in the subject area and have good judgement about the quality of research, and convince those people to review a submitted manuscript. Really hard.
Because it’s hard to find good reviewers, their time and effort are a finite and precious commodity, whether or not their time is compensated. A reviewer who is asked to review a weak paper always has to decide whether that is worth her time. Other things being equal, it’s easier to get a ‘yes’ to that request if the authors are known to be junior people who would benefit from help and advice. It’s more likely (and perfectly reasonable) to get a ‘no’ if the author is an experienced old fart who should know better.
Paying for peer review would be a benefit. Other systems include post-publication peer review.
A sort of hybrid system involves authors uploading manuscripts to an unreviewed open-access online repository. Readers can critique those articles. Editors then scan the articles and the critiques to find articles they think might be suitable for their journal, and put them into anonymous review. Elements of this are already in use.
All of these alternatives are better than the direction we’re headed, where all research products are uploaded without review to a single universal unfiltered open-access repository. Research Twitter but with 280,000 characters, and no quality control (but also no discrimination or gatekeeping). AKA the heat death of the academic publishing universe.
I exaggerate of course, but only a little.
I see no reason why scientific journals shouldn’t strive for diversity — perhaps by inviting or soliciting articles and reviews from groups whose contributions have been historically underrepresented — so long as doing so does NOT result in an adverse impact on the quality of the journals’ output.
IOW, the journals might endeavor to broaden their horizons without lowering their standards when striving to attain such a goal.
I believe (without ever having been an editor for any of them) that for the prestigious journals the situation is more like it is for prestigious colleges – more quality applicants than you have spots. So they probably don’t do any “horizon looking” at all, because they’re already overwhelmed with good science entries. What’s the time from acceptance to publication up to for Science these days – a eighteen months? Two years? Something ridiculous. Going out and looking for additional submissions would make it worse.
So, should they consider the race (sex, etc.) of the authors when deciding which to accept, as a way of increasing diversity? Or is being blind to the race (sex etc.) of the author better? And if we do the former, haven’t we just incentivized unethical conduct by rewarding people who game the system by adding unnecessary authors or rearranging the author list?
Lots of questions. Lots of people all trying to do the right thing but disagreeing on the best way to achieve it.
And I think I may have posted too much on the topic contra da roolz, so I’ll pause until tomorrow. Apologies to anyone wishing to continue the conversation in the near term.
I’ll confess ignorance on this one, Your Honor. 🙂
If 2+2=4, no. If 2+2=5, then yes.
Since my 2008 retirement, I am a decade removed from the peer review process, but srongly endorse jerry’s idea of author-blind reviews. Whether or not it leads to diversity improvement, i do believe that it would lead to better final papers in that all information to make the paper stand alone would more likely be presentedin the paper. By this i mean that i am sure that at least i sometimes would impute meaning to a paragraph because i knew what the author meant to say and maybe not always call him (yep in my field of aerospace enginineering, it was pretty much a 10:1 odds on bet that the author was a him) out for a better or more complete presentation. In any case i think that the result of blind authorship would be what my math friends call positive semi definite – either better or at worst the same, but certainly not worse than author-known. In my field the reviewers were blind to the author with the journal associate editor knowing both and mediating comments if required.
“[B]ut minus Asians since they are surely overrepresented among authors” – that reminded me of a plaintive tweet reproduced in Times Higher Education some years ago in relation to the growing number of authors listed for some science papers (e.g. from CERN’s Large Hadron Collider): “No, I’m not the 37th listed author, I’m the 37th listed author called Wang!”
Blind processes for both authors and reviewers sounds like the way to go. Diversity in academia is desirable to the extent that it reflects equality of educational outcomes, ambition, and opportunity. But it won’t influence the scientific output of journals, surely?
The BBC had an annoying piece about the low acceptance rates for black PhD candidates, failing to specify (as far as I could see) whether the issue was 1) black students make up a disproportionately low percentage of overall PhD students because fewer apply or 2) those black applicants who do apply are less likely to be accepted than their white counterparts. https://www.bbc.co.uk/news/education-54934953
It seems black students achieve lower undergraduate degree outcomes, which certainly needs addressing. Until it is, then problems further along the academic pipeline are inevitable.
Great posting, and I totally agree with you. You cannot always have “perfect representation” of ethnic groups as they appear in the population. There are various reasons, including different education levels, cultural factors that promote interests in different fields, that come into play. Part of the reason for some of the gender gap is simply due to a difference in interests. Fewer women become mechanics because fewer of them are interested in cars. Fewer men become nurses partly because of the over-emphasis on “caring and nurturing” that might be uncomfortable for some men. This is not the problem. The problem would be a hypothetical situation in which some races/genders etc are kept out simply for being who they are. This is NOT THE CASE, and in fact huge efforts are undertaken to recruit under represented groups.
The truth is, that the emotional/hysterical left is simply more outspoken about throwing tantrums when they don’t get what they like. The other sides simply remain silent, creating the false impression that they are outnumbered. Free speech repression by big tech also comes into play here. The problem is multi-fold.
Off-topic:
… second notice : off-topic. Danger of intellectual pollution— my comment really does not apply at all to this post, and I apologize but I cannot resist the urge to write this out:
The comments on the musician audition post are closed. That story bugged the hell out of me so I refrained from comment. I went and read through, I’m not sure anyone pointed this out about that story.
Music performance is intrinsically about
the magic spell of watching and seeing people emote through wood, steel, wind, air — inanimate objects — and watching the fingers or other parts of the body, including eyes, as they bare the soul of the music. Furthermore, music is intrinsically about the lands where the composers worked — be it Poland (Chopin), Russia (Tchaikovsky), and so on. If a screen is to serve a purpose, I’d think it has to be balanced with screenless performance — which might have been mentioned in the original piece. So much of the point of music is lost by watching a performance through a screen.
Ok, thats all, I feel better now — thank you.
“First, that a paucity of diversity—which of course means ethnic diversity, but minus Asians since they are surely overrepresented among authors…”
Don’t forget Jews, the always-ignored (or hated on) minority for the Woke. It’s not as if many of us aren’t apparently Jewish, just as someone from Asia is apparently Asian. People could tell from my looks growing up that I was Jewish, and I was bullied regularly for it. Plus, Jews suffer from the most hate crimes per capita, and it’s not even close. But I digress..
With all this talk of how a lack of diversity is automatic evidence of prejudice, I wonder, will these same people ever say that, for example, college is sexist against men? After all, women now earn the majority of college degrees, and have an even greater advantage as degree level rises, and this advantage has been increasing yearly for quite awhile now. They also dominate many fields, like psychology (according to government statistics, they earned 75% if psychology degrees at all levels). According to the Washington Post, there are now more women in the workforce with college degrees than men (Pew Research calls this a “milestone”).
Of course, I don’t think this shows prejudice, but, according to woke calculus, it’s prima facie evidence of sexism in college education.
I am shocked, shocked by the lack of diversity, equity, and inclusion in Sports. In basketball, people of short (or even average) stature are drastically under-represented. In football, players with a slight build are far fewer than their frequency in the general population. And in track events, slow people are not represented
at all. This rampant, systemic ableism must be confronted, perhaps by subjecting everybody to mandatory anti-ableism training workshops. My own Consulting company is prepared to set up these workshops—a public service which, as it happens, will not be cheap.
CRT is faith based and deductive, science is evidence based and inductive, and never shall the twain meet. 500 years ago there was an expectation that science ‘had to’ line up with the Bible if it was correct, now it ‘has to’ line up with CRT. There appears to be an extreme tenacity in this ideological instinct among us humans.
In my field (Statistics) the usual system is
authors – names visible
referees – blinded
authors do *not* recommened referees, AEs choose the referees
A few journals blind the authors as well, and I see no reason not to do this.
I doubt it will make much difference, and as noted above you can often gues the author or referee anyway, but why not do it?
Jerry said: ‘while I believe in affirmative action, I think it has to stop at some point in the hierarchy.’
I think it should stop wherever it leads to lesser-qualified people in positions where this matters, nurses and firemen come to mind. Also, I think that the people who politically and legally support quota-like affirmative action in all areas should not be exempt and get a full taste of their own remedy. E.g., affirmative action should be the rule for the legal assistants of Supreme Court judges who support such measures, for staffers in congress.
Affirmative action is demeanig to members of underrepresented groups, think of a black or female physicist assistant prof whose students all wonder whether he/she got the job for his race/gender.