There’s been a lot of press about a study just published in Science in which a large consortium of researchers tried to replicate 100 studies published in psychology journals, and managed to get significant results in only 36% of the replications. Further, investigators who repeated the earlier studies judged subjectively that they had replicated the original results only 39% of the time.
There has been a lot of analysis of these results in the press, with conclusions ranging from “psychology can’t be trusted” to “this is just normal science.” I’ll talk more about that, and summarize the Science paper, tomorrow, as it’s complicated and I need to read it for the third time. But in the meantime, and to save space in tomorrow’s post, I’ll cite one study that failed to replicate when redone by the large group of investigators constituting the “Open Science Collaboration” (OSC). That study is of special interest to me because it involves free will and its supposedly salubrious effects on society.
The paper chosen was from Psychological Science, one of the three well-known journals chosen by the OSC as sources of the 100 replicated studies. And this one was Vohs and Schooler’s 2008 paper (reference below) which I discussed in 2014, reproducing its abstract from the journal:
Does moral behavior draw on a belief in free will? Two experiments examined whether inducing participants to believe that human behavior is predetermined would encourage cheating. In Experiment 1, participants read either text that encouraged a belief in determinism (i.e., that portrayed behavior as the consequence of environmental and genetic factors) or neutral text. Exposure to the deterministic message increased cheating on a task in which participants could passively allow a flawed computer program to reveal answers to mathematical problems that they had been instructed to solve themselves. Moreover, increased cheating behavior was mediated by decreased belief in free will. In Experiment 2, participants who read deterministic statements cheated by overpaying themselves for performance on a cognitive task; participants who read statements endorsing free will did not. These findings suggest that the debate over free will has societal, as well as scientific and theoretical, implications.
Since then the study has been touted widely (see below), often as proving that determinism is bad for society. It’s has been used—by Dan Dennett and Eddy Nahmias among others—to show that unless people believe in some form of free will, they’ll behave badly and society will fall apart. I find that argument very odd, for when a similar argument is applied to God (“It’s important for society to be religious, for without religion, the moral glue that keeps us harmonious will dissolve”), it’s rejected by people like Dennett and me. In this way, belief in free will has come to resemble “belief in belief” (as Dan calls it) in a religious sense. But why accept one argument for societal harmony but reject the other?
But as a friend wrote me about this study and how it’s used to promote free-will compatibilism:
A huge proportion of factual statements could, when read immediately prior to an opportunity to act, nudge the percentage of alternative behaviors up or down by a few points (particularly in experiments where the subjects can infer that the statement was there for a reason and the point of the experiment is to see what they’ll do in response – but let’s put that aside). The response isn’t to disseminate falsehoods that will nudge people to behave better; it’s to disseminate explanations as to why one shouldn’t behave badly even though certain propositions are true.
That last sentence is full of wisdom, and we should remember it whenever we feel tempted to foster “belief in belief”—whether that belief be in God or free will.
At any rate, the New York Times, in a subsection of its blurb on the OSC paper, describes three famous studies that were not replicated but were still widely cited in the press. You can see them all at the piece called “Three popular psychologies studies that did not hold up”. I’ll reproduce only what the Times says about the Vohs and Schooler study.
Free Will and Cheating:
In 2008, a paper in Psychological Science found that people were more likely to cheat on a test after they had read an essay arguing that behavior was predetermined by environmental factors. The authors suggested from their findings that belief in free will had societal implications.
The redone study found an effect pointing in the same direction as the original, but far weaker. One possible reason, the authors suggest, had to do with how subjects’ opinions about free will were manipulated. Participants read an essay, and it’s plausible that they were not as engaged in reading and thinking about it as were those in the first study.
The study was cited 341 times in other journals, the most of any of the 100 studies that the Reproducibility Project tried to replicate. There are 24 citations listed in the PubMed database.
In popular news media at the time, the study was covered with a focus on what it meant for societal belief in free will. A Scientific American report in August 2008 called the study clever and added, “The results were clear: Those who read the anti-free-will text cheated more often!” In Psychology Today in March 2008, a reporter wrote, “Reducing belief in free will might also make people exercise less and drink more.” A New York Times story in February of the same year said that the researchers interpreted their findings to raise, “questions about how human behavior might change if the belief in free will continued to decrease.” However, it added that the researchers, “cautioned against reading too much into the results.”
I’m amazed—and appalled—that the Vohs and Schooler study was not only cited uncritically (doesn’t anybody care if there are long-term effects?), but was even cited improperly, as if denial of fee will would affect exercise and drinking.
I’ve also written that Rolf Zwaan at the University of Rotterdam has failed to replicate the Vohs and Schooler result, so that makes two failures to replicate, even if you accept that some truth is conveyed in the kind of experiment that was done originally. (I have serious doubts about whether cheating immediately after reading a deterministic passage—even if it’s a real effect—says anything about long-term behavior.) All I know is that I haven’t been tempted to cheat more often since I’ve become a hidebound determinist and incompatibilist!
But in the end, these are the words to remember: “The response isn’t to disseminate falsehoods that will nudge people to behave better; it’s to disseminate explanations as to why one shouldn’t behave badly even though certain propositions are true.”
Vohs, K. D., and J. W. Schooler. 2008. The value of believing in free will: encouraging a belief in determinism increases cheating. Psychol. Sci. 19:49-54.