Posting will probably be lighter than usual (even during my travels) over the next ten days, as I’m weary of arising at 4:30 to write posts, and, more important, I’m off to the Big Island soon for traveling and snorkeling. I’ll try to keep the Hili dialogues going, although it’s tough without Grania, but bear with me until July 9.
Today’s Execrable Science Story comes from Bloomberg, which you can access by clicking on the headline below:
As the article reports, 11 years ago Science published a paper by a group of psychologists that, says Bloomberg, claimed to find biological differences between liberals and conservatives. The paper is this one, and you can get a free pdf by clicking on the screenshot:
From Bloomberg:
According to the paper, conservatives tended to react more to “sudden noises” and “threatening visual images.” This result, which suggests that political liberalism and conservatism spring from deep, indelible sources rather than reactions to the issues of the day, suggests that polarization will never end — that the populace will always be divided into two camps, separated by a gulf of biology.
This is the kind of paper that demands replication, and indeed, when two groups tried to replicate the results, they failed:
Fast forward a decade, though, and the claim is unraveling. In a working paper published this month, another team of psychologists attempted to repeat the experiment, and also conducted other similar experiments. They failed to find any evidence linking physical-threat perception with political ideology. But when they tried to publish their paper, Science desk-rejected it — that is, the editors refused to even send the paper out for peer review, claiming that the replication study simply wasn’t noteworthy enough to be published in a top journal. Meanwhile, another team of researchers also recently tried to replicate the original study, and failed. So even though at this point the evidence proving a biological basis for liberalism and conservatism seems to have been invalidated, it’s unclear whether this fact will make it into the public conversation.
The “desk-rejected” link takes you to a Slate article by all four of the rejected paper’s authors, who provide a good justification for publishing their paper. But it wasn’t even reviewed by Science! As the authors say in Slate:
We did not expect Science to immediately publish the paper, but because our findings cast doubt on an influential study published in its pages, we thought the editorial team would at least send it out for peer review.
It did not. About a week later, we received a summary rejection with the explanation that the Science advisory board of academics and editorial team felt that since the publication of this article the field has moved on and that, while they concluded that we had offered a conclusive replication of the original study, it would be better suited for a less visible subfield journal.
We wrote back asking them to consider at least sending our work out for review. (They could still reject it if the reviewers found fatal flaws in our replications.) We argued that the original article continues to be highly influential and is often featured in popular science pieces in the lay media (for instance, here, here, here, and here), where the research is translated into a claim that physiology allows one to predict liberals and conservatives with a high degree of accuracy. We believe that Science has a responsibility to set the record straight in the same way that a newspaper does when it publishes something that doesn’t stand up to scrutiny. We were rebuffed without a reason and with a vague suggestion that the journal’s policy on handling replications might change at some point in the future.
There’s not much more to add to this. Science has behaved shamefully, with its usual hauteur and diffidence. They are a bunch of self-iimportant gits. They won’t, I suspect, even add a note to the original paper that it had failed replication; and that is the minimal thing to do to call attention to the issue.
And what if the failures to replicate don’t get published? (I hope and expect that they will.). For science to progress, false or questionable results must be put into the public domain, for that’s the only way to ensure that the truth will out. By allowing a possibly incorrect result to stand uncorrected, Science is degrading the entire scientific enterprise. Their behavior is reprehensible.
I should add that, according to the replicators, the paper’s original authors were supportive of the efforts to publish the failure of replication. It’s the damn journal—I call it a “magazine”—that’s to blame here.
Shocking – “reprehensible” behaviour indeed!
Damn, sounds like you’re fixing to make Oliver Twist-style orphans of us all. Hope you make it back stateside before we’re put to hard labor as chimney sweeps or fall in with the petty-larceny of Fagin. 🙂
The prestige journals appear to be interested in just that – prestige, and not serving the cause of scientific discovery and accurately informing the scientific community and public. If they were, they would not emphasis novelty of results over properly performed experiments (adequate sample sizes, etc.) nor would they eschew replicate studies.
>The prestige journals appear to be interested in just that – prestige, and not serving the cause of scientific discovery
Unfortunately, the same can be said of many scientists and universities. It’s one of the reasons why I left academia – I was continually pushed to produce the “right” results (i.e. results that would be published in a top journal), even if said results were not reliable.
Negative and disconfirmatory results were ignored as “failed experiments”, multiple things were looked into and only the “interesting” results reported, and experiments were designed to produce the results that confirmed the investigators’ theories, rather than trying to disprove those theories (which is what a good scientist should try to do).
None of this was done consciously or deliberately – it all just naturally came about from the massive pressure to publish, to get funding, and to get promotions. Good people in a bad environment led to bad science.
Yes I agree. This problem is much broader than Science-the-Journal, it’s as broad as Science-the-discipline. Reproducing someone else’s experiment doesn’t (as a generalization) get you as good a publication, doesn’t looks as good on your resume for tenure (as original research), and doesn’t look as good to granting agencies. If Science-the-discipline really wants to value reproduction, all of those things have to change.
sub
Do not worry about the place here. It will still be here when you get back. Go to Kailua-Kona and have more fun. Less people and more land.
Thanks. Enjoy Island
Replication studies are important
I don’t think journals are responsible for all follow-up studies of everything they publish. Suppose there are 6 follow-ups, 5 replicate, and 1 does not. Do they publish the failure when it is submitted? Do they monitor whether there are positive results? Will they only publish failures or also replications? Putting the onus on journals is unreasonable. Researchers should be responsible for literature reviews that identify failures to replicate.
An interesting point, but I think the matter should be considered on a case by case basis. Sometimes the situation would be highly significant.
I know I’m not alone in noting that I’ve really only read Science for the want ads, and then only briefly!
I can certainly see why someone would question the findings of the earlier study. For example, why would someone who reacts more strongly to loud noises and threatening images be opposed to gun control? Barring an axe to grind (a threatening image?), that makes no sense whatsoever.
As a left-leaning individual in a painfully red place (Indiana) perhaps I can provide an answer, and explain what I think is the correlation.
The most common responses I hear whenever I argue about gun control are these: “I NEED MY GUNS TO PROTECT MYSELF” and “I WON’T BE SAFE WITHOUT MY GUNS”
I see pro-NRA bullshit all the time where I live. The message is always the same: “BUY GUNS, YOU WON’T BE SAFE WITHOUT THEM”
I just recently saw an ad for a military style rifle advertising it as a “great tool for home defense”.
So, if your someone who reacts strongly to scary images and loud noises, and you see an NRA ad showing some violent protest in the middle east with the description “buy a gun to protect yourself”, what are you going to do?
I first started down this line of thinking after watching a conservative neighbor (who already owned a few guns) go out and buy another gun after he watched some scary slasher flick.
This was an unfortunate decision by Science, especially considering how much media attention the original study received. More generally, the prestige of the journals scientists publish in is considered when they apply for grants and for jobs. Thus the idea that replication studies are not “novel” or “exciting” enough to be published in prestigious journals can discourage scientists from carrying them out. I can see why Science and other high ranking journals would prefer to publish new findings over replications. However replication is an essential part of science, and I think they need to publish replications of high profile studies, not only to correct the record on studies that could not be replicated, but to encourage scientists to undertake replication projects.
Hmm, I wonder if Nature, which definitely competes with Science as a top tier journal, would publish it. There would be a rather satisfying undertone of dissing ‘that other journal’in doing so.
:)!
They probably won’t. They are well aware that they are vulnerable to a pissing-war breaking out. As is pretty much every journal out there.
I was thinking – rejected non-replication papers might find a natural home at Annals of Improbable Research who can poke fun – justifiably – at the “replication problem” without much fear of retribution. The old “the Jester is the only one who can tell the Emperor that he is naked” situation.
I’d long suspected this claim’s validity, though I’ve seen it circulate widely. Given my own life experience and so many others like me who’ve swung from conservatism to liberalism, the notion that it is deeply engrained seems suspect. Couple that with the regional differences in political attitudes, and the problems with the idea only get harder to defend.
If you are going to go the “my life experiences” route, then I would like to state that my life experiences have almost completely validated this theory.
The problem that I see is that many on the left don’t like the idea, much in the same way that those on the right don’t like the idea of global warming.
Does this Pinker tweet endorse the idea that genetics influence the conservative/liberal bifurcation?
https://twitter.com/sapinker/status/1135295172047192064
This particular failure to replicate doesn’t invalidate the general finding that politcal viewpoints are highly correlated with personality, and that personality is highly heritable.
Indeed.
There have been quite a few studies, both before and after the 2008 study, that come to that same general conclusion.
The way the author declares “the world should know there is no difference..” is incredibly dishonest.
It’s very similar to when religious nuts claim evolution isn’t true because some ancient skeleton was misdated or some other error was discovered.
Maybe a “Journal of Failed Replications” should be created?
I’m sure such a debunking journal would be wildly popular. I, for one, would be referring to it on a regular basis.
http://www.improbable.com/ ?
Some of us (like me, who has precisely 0 power to make it happen) have been calling for that for a long time.
Unfortunately its all part of the “crisis of replication”, desk drawer effect etc etc.
Feynman’s famous “cargo cult science” lecture drew attention to psychology’s lack of interest in followup/replication studies, and it hasn’t improved much since then.
However, given believable probabilities of false positives, test power and choices of whether to publish or not, we would only expect maybe a third to a half of studies to “replicate” (i.e. positive both times) anyway.
On a related note, nice article here on a Sokaling of Nutrition Bollocks:
https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800
with the published paper here
https://www.medicalresearchjournal.org/index.php/GJMR/article/view/922/832
I’m sorry to have to say this, but that’s nonsense. You can’t simultaneously have a “replication crisis” and a “file drawwer effect”. You are going to need to sort out your prejudices about psychology one way or the other. Either psychologists are obsessivley trying (and failing) to replicate studies and thus those studes are beign falsified, or they are sitting on such failures and failing to report them. Pick one.
Incidentally, calling things a “replication crisis” shows an utter ignorance of the way science works. “Bold conjectures, conclusive refutation”, as Popper (rightly) had it. We are discovering some things in psychology that are wrong. Thats a cause for rejoicing. Just as my pointing out that you are wrong should cause you to rejoice. You are now less wrong. Hooray
Not sure what you mean by this. “Crisis of replication” and “file drawer effect” are both well-known features, although there is certainly plenty of argument to be had over exactly how much of a problem there is in both cases.
The “FDE” is almost certainly part of the “CoR”, in the sense that “positive” results get published in Science whereas “negative” results get published in less prestigious outlets or not at all. This applies to studies before the “positive” one, probably more so than those after it. If enough groups study the same (in fact non-existent) effect then eventually one of them will get a (false) positive and get published. It shouldn’t be a surprise when if another group then tries to replicate the positive results that it doesn’t do so.
I agree that the “crisis of replication” is probably overblown (hence the inverted commas). If you run a lot of experiments where variance is high (e.g. people are involved) and likely effect sizes are small then of course you will get lots of false positives. One of the recent papers on this (I’m sure you know the one I mean) had something like 30 out of 100 studies “replicating”, but IIRC others pointed out that most of the results in the followups were still within the prediction intervals of the original studies.
This is the paper which noted that we wouldn’t expect all studies to “replicate” anyway:
https://www.tandfonline.com/doi/full/10.1080/00031305.2018.1543137