“Peer Review”: A new science journal that critiques articles in other journals

Luana sent me this tweet, which I’d missed, announcing the founding of a new scientific journal that exists to critique articles in other journals (with the original author given the right of reply). Click on the screenshot if you want to go to the original tweet:

To see the Wall Street Journal op-ed by Kevin McCaffree and Colin explaining the journal, click on the screenshot below—or you can find the article archived here.

Some excerpts:

We’re often told that science is “self-correcting.” But science isn’t like a thermostat regulating your home’s temperature. It’s a human institution run by fallible human beings. Scientists and scholars are susceptible to career incentives, moral fads, groupthink and fear. When those pressures capture journals or entire fields, peer review can become less a filter for error than a credentialing system for fashionable nonsense.

. . . Decades of studies on publication bias, replication failures and political bias in the social sciences have shown that peer-reviewed papers are often less reliable than the public assumes. John Ioannidis’s famous 2005 paper, “Why Most Published Research Findings Are False,” remains disturbing because its basic insight about the fallibility of medical research remains true. In fields that rely heavily on narrative or qualitative methods, or that touch on politicized topics (as much social science does), ideology influences which questions are asked and which conclusions are professionally acceptable.

The authors then mention the Sokal hoax as well as the “Grievance Studies affair” involving submission of bogus papers to social-science and humanities journals by Helen Pluckrose, James Lindsay, and Peter Boghassian. (There were no submissions to STEM journals in either “affair” unless you consider gender studies journals as being in STEM.

A bit more:

This problem is growing more serious. Across swaths of the humanities, social sciences, medicine and biology, some narratives have become taboo. Papers presenting contrary evidence or dissenting viewpoints are rejected without comment. Letters to the editor, which are supposed to provide a quick way to respond to flawed work, are ignored or unavailable. The result is an ideologically biased literature that’s presented as an expert consensus and cited by journalists, courts, school boards, medical associations, government agencies and lawmakers to justify policies that affect millions of people.

The most obvious answer is better peer review. But ideologically captured fields have little incentive to correct themselves. As a result, objections to progressive orthodoxy are relegated to social-media threads, blog posts and newspaper opinion sections.

This is where the myth of “self-correcting” science becomes a problem. People assume the system will fix itself, but first someone has to notice the problem and create a mechanism for correction.

That is what we have done. As an editor-in-chief and a member of the editorial advisory board of Theory and Society, an interdisciplinary journal published by Springer Nature, we are proud to announce a first-of-its-kind article type called “Peer Review.” The purpose is to avoid procedural traps that can prevent legitimate criticism from being published and to recover what peer review was supposed to be: serious, good-faith analysis by experts seeking clarity and truth.

As in postpublication peer review, a Peer Review article may address a paper from any scholarly journal so long as it raises concerns about methods, evidence, logic, definitions or theory. The focus must be on claims, arguments and scholarly standards, not the author’s character or motives.

Submissions, limited to 2,500 words, will undergo a simple merit review rather than endless rounds of gatekeeping. An editor or subject-matter expert will ask a straightforward question: Is this critique coherent, serious and reasonable enough to deserve scholarly attention? If so, it will be accepted.

This is a good idea, and I can easily see myself writing a short response to some pieces that I find deficient. (Some of my website critiques of “sex-is-a-spectrum” posts might have been appropriate.

The only problem is what to do with papers (not just critiques) that try to air subjects that are forbidden or inflammatory.  Those might be suitable for The Journal of Controversial Ideas, but I’ve never seen a straight science/data paper there. (Granted, I haven’t looked at every issue.)

Anyway, pass this news along to those who might be interested.