Okay, not really on psychic powers, but on precognition. Last October I posted about Daryl Bem’s paper in the Journal of Personality and Social Psychology, “Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect,” (download a preprint on his webpage). I summarized its results as follows:
The paper purports to show that a choice that you make in a computer test can be influenced by stimuli you receive after you’ve already made the choice. This implies you have some way, consciously or unconsciously, of detecting things that haven’t yet happened.
I also mentioned some criticisms of these results by others; and many of the hundred-odd comments were also critical.
Yesterday’ss New York Times reports a strong backlash by scientists about the paper.
“It’s craziness, pure craziness. I can’t believe a major journal is allowing this work in,” Ray Hyman, an emeritus professor of psychology at the University Oregon and longtime critic of ESP research, said. “I think it’s just an embarrassment for the entire field.”
The editor of the journal, Charles Judd, a psychologist at the University of Colorado, said the paper went through the journal’s regular review process. “Four reviewers made comments on the manuscript,” he said, “and these are very trusted people.”
All four decided that the paper met the journal’s editorial standards, Dr. Judd added, even though “there was no mechanism by which we could understand the results.”
But many experts say that is precisely the problem. Claims that defy almost every law of science are by definition extraordinary and thus require extraordinary evidence. Neglecting to take this into account — as conventional social science analyses do — makes many findings look far more significant than they really are, these experts say.
“Several top journals publish results only when these appear to support a hypothesis that is counterintuitive or attention-grabbing,” Eric-Jan Wagenmakers, a psychologist at the University of Amsterdam, wrote by e-mail. “But such a hypothesis probably constitutes an extraordinary claim, and it should undergo more scrutiny before it is allowed to enter the field.”
Dr. Wagenmakers is co-author of a rebuttal to the ESP paper that is scheduled to appear in the same issue of the journal.
Wagenmakers commented on my post and gave a reference to his own paper rebutting that of Bems.
The report goes on:
Peer review is usually an anonymous process, with authors and reviewers unknown to one another. But all four reviewers of this paper were social psychologists, and all would have known whose work they were checking and would have been responsive to the way it was reasoned.
Perhaps more important, none were topflight statisticians. “The problem was that this paper was treated like any other,” said an editor at the journal, Laura King, a psychologist at the University of Missouri. “And it wasn’t.” . .
. . . So far, at least three efforts to replicate the experiments have failed. But more are in the works, Dr. Bem said, adding, “I have received hundreds of requests for the materials” to conduct studies.
I’ll bet big bucks that the effect vanishes, simply on the grounds that a mechanism for precognition seems unlikely. Perhaps the paper did deserve more thorough reviewing given its controversial nature, but the truth will out. And in the the advocates of precognition will be on even weaker ground.
p = .05 —> Nineteen of twenty experiments fail, then you get to publish the 20th.
I can’t think of a better summary.
b&
Perhaps James Randi or his foundation should serve as mercenary peer review checkers. There is a big difference between accepting a scientific paper as if all the information is correct, and analyzing the experiments for unreported fraud. It reminds me of the homeopathy ‘proving’ paper in the 80s, where once they went in and looked at the lab, they found consistent, continual, and knowing manipulation of both the data and the experiment. I would not be surprised if it was the case here, but you’re not going to find that out in your office looking at a pdf.
Do you mean Benveniste’s work? Although some people accused Benveniste of fraud, I’ve not seen a report in which that was substantiated – have you? Guilty of wishful thinking and extreme sloppiness, yes, but your accusation goes a lot further than that.
Perhaps I was being a bit mean, but that guy’s work was shoddy to the point of being suspiciously so, at least in my opinion. Salaries for some of his workers came from a French homeopathy company too. Certainly he doesn’t deserve to work in science anymore without doing some sort of major mea culpa.
Here’s the original paper http://www.criticandokardec.com.br/benveniste02.pdf
Last October I posted about Daryl Bem’s paper in the Journal of Personality and Social Psychology…
This research has received considerable press already for something that is still months away from publication.
I’ll bet big bucks that the effect vanishes…
I’ll bet big bucks that if the paper is published in a reputable journal, it will be cited by ESP proponents for decades, perhaps centuries, regardless of whether it is discredited or not. Benveniste & homeopathy, anyone?
“But such a hypothesis probably constitutes an extraordinary claim, and it should undergo more scrutiny before it is allowed to enter the field.”
No…
How about simply publishing with an editorial commentary? I would like to read preliminary results without having to attend conferences or having to know everyone in the field…
How about “The Journal of Preliminary Results”
Sounds suspiciously like the ‘Journal of Irreproducible Results’ to me.
Two things:
1. It should be the responsibility of the researcher to clearly indicate whether we are talking about preliminary results, not the editor. It should also be the responsibility of the journalists reporting on it to be clear about this. However, for this paper, it has been claimed to offer confirmation of precognition, not preliminary results. Whether we should blame this on the author or on the reporting is another issue. However, I will note that one of the main criticisms from the Wagenmakers et al paper is that Bem has not been clear whether he was on a fishing expedition or trying to confirm a hypothesis.
2. Psi has had more than its chance to offer preliminary results. If there were anything to it, you should be able to demand more by now.
“How about “The Journal of Preliminary Results””
Great. Here is your golden opportunity to establish, fund, administer & publish such a journal!
The only reason that paper was accepted was because it was written by Bem, and the reviewers gave it coddling reviews and the editor gave it halo treatment. Nobody paid attention to the fact that it’s implausible, and the “findings” are certainly mistakes in the design, analyses, and data cleaning–to be generous and omit the other possibility–that they are contrived.
“Psychic” paper provokes backlash
You told them one too many times you were a time-travelling safety inspector?
I disagree with the skeptics in that article. From everything I read, the experiment seems to have been well done. This is exactly what the scienceophiles have been asking the paranormal people to do all along – set up good experimental conditions and test their ideas.
It just seems to me that the standards for establishing paranormal effects should be the same as turning over any current paradigm. Treat them all fairly and let the data speak for itself.
Yes, the true believers will annoyingly trumpet these preliminary results all over the Intertubes, but we’ll just have to deal with that.
Exactly, but turning over a paradigm is not the same as doing standard research within a paradigm, and thus the standards are not the same. As the saying goes, extraordinary claims demand extraordinary evidence. This paper does not provide extraordinary evidence.
On the other hand, failure to provide whatever standard of evidence is deemed “extraordinary” is not failure to provide evidence, so I don’t see why it need damn Bem’s paper to rejection.
The problem is that there is extraordinary evidence that precognition cannot exist. That evidence is our current understanding of reality.
So what is more likely, that our current understanding of reality is wrong, or that Bem got an anomalous result? And why would a major scientific journal publish such a result? Certainly not because of the science.
The paper doesn’t provide extraordinary evidence (not even close), but it is some effect these guys saw, which they documented well in a way that allows others to try to replicate their experiment. That’s exactly the kind of thing that journals should publish IMHO.
Don’t squash the publication of one of the very few times an experiment was designed well and got some positive results. That’s what the paranormalist folks always accuse the science “establishment” of doing. Be open, and let the paranormal stand or fall on its results.
I endorse this message, with the caveat that if it’s just anomalous there’s lots of odd anomalies floating about.
But whether this was a good experimental setup is exactly the matter in dispute here.
It’s craziness, pure craziness. I can’t believe a major journal is allowing this work in,” Ray Hyman, an emeritus professor of psychology at the University Oregon and longtime critic of ESP research, said. “I think it’s just an embarrassment for the entire field.
I don’t know why Hyman’s being so presumptive when he’s admitted that studies into ESP – the ganzfeld experiments, say, or Targ and Puthoff’s work into remote viewing – is compelling, if not quite convincing.
Darren,
Nobody paid attention to the fact that it’s implausible, and the “findings” are certainly mistakes in the design, analyses, and data cleaning–to be generous and omit the other possibility–that they are contrived.
While I’d be an idiot and an arse to claim it won’t be so, isn’t scepticism about judging claims after the fact?
Would I have to be a complete cynic to point out the lengthy history of fraud by both experimental subjects and experimenters in the field of ESP research?
No. Would it be selective not to note that several of them were exposed by their own colleagues (Walter Levy, for example, who was confronted by J.B. Rhine)? Perhaps. Would it be irrational to assume that if some were dishonest/incompetent they all must be? Yes.
I would split the difference here. If it was good science, we can’t prejudge. It isn’t such a field, so we have to be skeptical.
Lack of mechanism is not a reason for holding back publication. It’s possible to discover a new phenomenon without understanding how it works.
In Demon Haunted World Sagan describes his refusal to sign a condemnatory statement on astrology which listed “lack of mechanism” among the reasons that astrology is bunk. He didn’t like that part and refused to sign it. Lack of mechanism is not the problem with astrology.
The precognition paper should have been held back pending a set of independently reproduced results. That the results are not reproducible is what makes it a non-phenomenon, not a lack of mechanism.
I disagree, with a caveat. Maybe you can argue that lack of mechanism cannot be the reason for holding back publication, but it certainly can be a reason.
Two words: Bayes’ Theorem.
The ol’ p < 0.05 standard is okay for trends with a plausible mechanism, but I would argue that as the standard does not take Bayes’ theorem into account, it does not hold up when your priors are very poor. A trend lacking a plausible mechanism — or even worse in this case, where it’s implausible that an unknown mechanism could even exist, since any such mechanism would apparently violate physical laws — demands an extremely high level of correlation to warrant publication. Otherwise, we can say with near certainty that (taking into account the priors) the result is a false positive of some kind.
Sure, one should publish new, standalone, observation. But when that is done in physics, the ante goes up. Instead of the usual 3 sigma for theories physicists often demand 5 or more sigma because it must be *really* unlikely it is a fluke.
Conversely I would argue that this isn’t “a new phenomenon”. Despite causality being a necessary and extremely well tested ingredient of physics (hmm, maybe *the one phenomena* better tested than evolution, come to think of it) people like to think that humans specifically can circumvent it.
And by their “soul” none the less, so dualism is rearing its ugly head to boot, that makes two severe strikes against. This is statistical fishing on an old and utterly unrealistic hypothesis.
Lack of mechanism can’t be the only reason for rejecting a claim. If there is no known mechanism, but the effect is strong, and repeatable, there would be no problem in getting your results published.
But it also works the other way around: if there is a plausible mechanism, it may help you predict specific effect sizes. Even if these predicted effect sizes are small, your prediction will help you design an experiment with enough power to reasonably conclusively confirm or reject the hypothesis, even with these small effects.
Without a mechanism, however, you can’t do this, and you can always claim that more research is needed. Or worse: you could come up with all sorts of ad-hoc explanations. Not having a mechanism means never having to admit you were wrong about one.
So in general I would agree with the harsh assessment, but I think there is one saving grace here that actually makes it worthwhile for this paper to have been published: Bem has been thorough, and more importantly, has been admirably transparent about the whole process. Transparency is always a good thing, but in this case it crucially redeems the value of publication. Let me explain.
I agree with the critics that it is a virtual certainty that Bem has not discovered real evidence of precognition. While stvs remarks in comment #1 that this could just be the 1 in 20 studies that gets p < 0.05 just by chance, there is some reason to believe that this was not the case, that in fact it was a flaw in the design of the study.
And that’s where the value of publication comes in. Bem’s thoroughness and transparency means that a) the presumed flaw(s) in the design of the study are not immediately obvious, and b) there is enough public information that the flaw(s) stand a good chance of being identified. That’s valuable, for all researchers.
As a study into social psychology or alleged precognition, this paper is as worthless as the critics claims. As a study into design of experiments and detection of experimental bias, it’s a potential gem.
He’s been transparent in publishing his data and source code, that’s true, and commendable. Where he has been less transparent is (as Wagenmakers et al point out) how extensive his fishing expedetion has been.
Perhaps the researchers who submitted the paper knew in advance their results would be treated with skepticism, which in itself proves the efficacy of their research. Heh, heh.
So, I have only two words to say about this, which in my mind show that science is working pretty much as it should and has for quite some time now.
…
cold fusion.
I predicted this.
I knew you were going to say that.
There’s another good reason to be suspicious, and that’s what Dr Coyne’s website is all about. If predictiing the future were a trait of some humans (and if it were heritable), then it’d be the blink of an eye in geological time before all humans had the ability, and knew they did. It’d be like vision and hearing; we’d be very aware of it.
And incidentally, humans would probably not be so gullible.
Or as the Wagenmakers paper points out, casinos wouldn’t be making a profit.
Is there any reason to presume it’d be all-inclusive? Under this logic I could reply to someone who’s asserted that humans have run 100m in ten seconds with, “Well, if we could why does it take me half an hour to get from my house to the bus station?”
My thinking was that precognition would produce such a huge fitness gain that (if it were heritable) it would spread through the population like wildfire. You’re probably correct that it’d be variable. In your analogy the trait is running, which we all can do, but to differing degrees. Even if it weren’t heritable, I think individuals who had the gift would surely notice that they did, and be able to demonstrate it convincingly.
In your analogy the trait is running, which we all can do, but to differing degrees.
True, but depending upon how powerful such an effect might be it wouldn’t need to be discernable. I see your point about survivability, though, and would be lying if I claimed to have an immediate response…
(While it proves nothing I was amused to see this story after reading Wagenmakers’ point.)
I haven’t read the paper in question, however my comment is in relation to such phrases regarding lack of known mechanism. Not understanding something doesn’t make it less true, it just means there is a gap in our knowledge: we don’t understand properly how gravity works.
If the experiment is correctly set up, (and I understand the criticisms that it wasn’t) then the fact that we can not explain (yet) the results does not invalidate the results.
Rejecting that which can be demonstrated, merely because “science says it’s untrue”, merely plays into the hands of Creationists, GW deniers and snake oil salesmen.
Sorry for double – just thought
Being a pre-cog in these circumstances may not a) be a massive evolutionary advantage, b) be a strong enough genetic trait.
For the sake of argument, lets say some people are pre-cog. if its a recessive gene then it could go hidden. Also the advantage confered doesn’t give a constant large advantage meaning those with it significantly outbreed non’s: In situation ‘A’ you die 98.5% of the time instead of 98.6% if you are pre-cog.
Also this is a very limited and formalised ability as described – in a test setting you receive specific information – you mind is not busy with the many things that it usually is, and the calm environment means the ability isn’t overloaded.