If you’re a scientist you’ll know this, and if you’re not, you should. The key to success as a research scientist is publishing papers in good journals, and the more papers the better. Ideally, as a biologist you’d publish in top-flight journals like Cell, Science, or Nature. Submitted papers are given to two or three anonymous reviewers who pass judgment on the paper, often deciding that it’s not good enough to be published (REJECTION), or might be published if some errors were fixed, new data analyses done, discussion modified, or additional experiments performed. On the basis of the reviewers’ takes, the editor decides whether to publish the paper as is (rare), reject it outright, or reconsider it if the referees’ objections were met. All of us have to surmount these hurdles.
The peer-review system is supposed to guarantee the quality of a paper, but we all know it’s fallible. For one thing, reviewers rarely have access to the original data, and even when they do they rarely redo the statistical analyses of the paper’s authors. We may spend a few hours reading a paper, but we have other things to do (like RESEARCH), and a few hours is rarely enough. According to Adam Mastroianni, a postdoctoral research scholar at Columbia Business School, the reviewer system, despite involving 15,000 years of effort per year by reviewers, has failed. He describes its failure in this article on his website Experimental History (free read, but subscribe if you read often). Click to read the paper:
Here’s why he thinks the review system hasn’t improved science:
Huge interventions should have huge effects. If you drop $100 million on a school system, for instance, hopefully it will be clear in the end that you made students better off. If you show up a few years later and you’re like, “hey so how did my $100 million help this school system” and everybody’s like “uhh well we’re not sure it actually did anything and also we’re all really mad at you now,” you’d be really upset and embarrassed. Similarly, if peer review improved science, that should be pretty obvious, and we should be pretty upset and embarrassed if it didn’t.
It didn’t. In all sorts of different fields, research productivity has been flat or declining for decades, and peer review doesn’t seem to have changed that trend. New ideas are failing to displace older ones. Many peer-reviewed findings don’t replicate, and most of them may be straight-up false. When you ask scientists to rate 20th century discoveries in physics, medicine, and chemistry that won Nobel Prizes, they say the ones that came out before peer review are just as good or even better than the ones that came out afterward. In fact, you can’t even ask them to rate the Nobel Prize-winning physics discoveries from the 1990s and 2000s because there aren’t enough of them.
Well, the flatness or decline of research productivity doesn’t say to me that review isn’t working, for there may be other social or economic factors affecting productivity. His link to new ideas “failing” to displace older ones goes to an article about how the incursion of novel ideas has slowed, and we’re “trapped in existing canons”. Again, that may have little to do with the reviewers of papers, and more to do with our gradually homing in on the truth. And of course new ideas have displaced older ones: the “neutral theory” in evolutionary biology is one of them.
But Mastroianni does have a point: reviewing is often hasty, sloppy, and unable to catch errors in papers. (He also cites the failure of much work to be replicated as a sign of the impotence of reviewers to stop bad science, but failures of replication can have many causes, including different populations or sample sizes, that have nothing to do with the prowess of reviewers.)
Where he makes his strongest point is citing studies where scientists run “hoax” studies in which they submit papers with deliberately added error. Those errors are caught only 25%-30% of the time. Also, if a paper is rejected or needs substantial revision, authors will often just send it to another journal (usually one that’s less selective), and eventually nearly everything can be published somewhere (there are 30,000 scientific journals!). But scientists are promoted and lauded not for publishing in low-quality journals.
So yes, the reviewer system is imperfect, often very imperfect, but what do we replace it with? We can’t just allow scientists to submit papers that aren’t even vetted, for then the journals, especially the very good ones, would be flooded with crap. So what is Mastroianni’s solution.
He doesn’t have one.
Here’s what he says:
What should we do now? Well, last month I published a paper, by which I mean I uploaded a PDF to the internet. I wrote it in normal language so anyone could understand it. I held nothing back—I even admitted that I forgot why I ran one of the studies. I put jokes in it because nobody could tell me not to. I uploaded all the materials, data, and code where everybody could see them. I figured I’d look like a total dummy and nobody would pay any attention, but at least I was having fun and doing what I thought was right.
Then, before I even told anyone about the paper, thousands of people found it, commented on it, and retweeted it.
. . .Total strangers emailed me thoughtful reviews. Tenured professors sent me ideas. NPR asked for an interview. The paper now has more views than the last peer-reviewed paper I published, which was in the prestigious Proceedings of the National Academy of Sciences. And I have a hunch far more people read this new paper all the way to the end, because the final few paragraphs got a lot of comments in particular. So I dunno, I guess that seems like a good way of doing it?
I don’t know what the future of science looks like. Maybe we’ll make interactive papers in the metaverse or we’ll download datasets into our heads or whisper our findings to each other on the dance floor of techno-raves. Whatever it is, it’ll be a lot better than what we’ve been doing for the past sixty years. And to get there, all we have to do is what we do best: experiment.
That’s not a good solution, as how do you find papers if they’re scattered all over the Internet? One way is to put your papers on the ArΧiv site, which doesn’t cover all fields, and let people have at them. Scott Aaronson agrees in part with Mastroianni, but in further FB comments he argues that a system of reviewers (and appeals) really does improve papers.
Although Mastroianni makes a good case for flaws in the current reviewing system, I don’t think he makes a persuasive case to get rid of it entirely. There must be a way to exercise some quality control over papers, or otherwise we’ll have to wade through gazillions of papers by loons and creationists to find what we want.
Winston Churchill is supposed to have said, “Democracy is the worst form of government – except for all the others that have been tried.” I think we can say the same thing about the scientific review system..