“The first principle is that you must not fool yourself — and you are the easiest person to fool.”
–Richard Feynman, “Cargo Cult Science”
The quote above is, to me, the best pithy characterization I know of the principles of scientific inquiry. Because we humans are ridden with confirmation bias—the tendency to try to affirm as true what we want to be true—science is structured to prevent such self-deception. Our culture of of replication, of peer review, and of pervasive doubt are all devices that have evolved over time to prevent scientists from fooling themselves. A famous example was the use of two completely independent teams of researchers to look for the Higgs Boson, researchers who didn’t know what each other had found until the very end of the experiments.
One of the best tools for scientific research is “blinding”: making sure that researchers, when making observations, are as far as possible kept in the dark about any information that could bias their results.
Here’s one example. When I was studying whether cuticular hydrocarbons in Drosophila (waxy substances that coat the fly’s body and prevent desiccation) can act as isolating barriers (different species have different hydrocarbons, and could potentially recognize each other as either same- or different-species mates since males “taste” females before mating), I transferred hydrocarbons among individuals by crowding them together in vials.By putting five females of one species into a vial with 50 females of a different but closely related species having different hydrocarbons, you can profoundly alter a female’s hydrocarbon profile, putting on her about half the “foreign” hydrocarbons from a different species. In other words, you can perfume females of a given species with hydrocarbons from females of another species that males don’t like to court or mate with.
After doing this perfuming, I then asked undergraduates to watch males courting both the perfumed females as well as control females belonging to the male’s own species (also crowded, but with members of their own species), and to record various aspects of male “courtship interest”, including circling, licking, wing extension, and attempted copulation (male curls abdomen under and jumps the female from behind). After each half-hour observation period, we took the female and measured her hydrocarbon profile using a gas chromatograph.
To ensure that any difference were due to hydrocarbons and not some behavioral change effected by crowding, we did the same thing with dead females that had been flash frozen in liquid nitrogen. That doesn’t change their hydrocarbons, and males (who aren’t particular about whom they court), readily court dead females, and even try to copulate with them.
It was crucial in these studies that the courtship observers didn’t know the identity of the target females, for that could have conditioned how they recorded or identified various behaviors. In other words, the observers were “blinded.” And the results we got were clear: the hydrocarbons that we predicted would turn off males—or turn them on when their own females’ hydrocarbons were put on foreign species— had a huge effect on male courtship in the predicted direction. The references are below, which include a nice paper in Science.
This kind of blinding is of course an important feature of medical studies. The gold standard for testing new drugs and therapies is the “double blind” study, in which neither doctor nor patient knows which treatment is being given. It’s common sense, really.
Sadly, many studies in ecology and evolutionary biology that could involve blinding protocols don’t. That, at least, is the conclusion of a paper in Frontiers in Ecology and Evolution published last May by Melissa Kardish and her colleagues (reference below; free download). They begin this very short (2.5-page) paper by noting the abysmal failure of many researchers to use blinding when appropriate and possible:
For example, a survey of kin-recognition studies—a cornerstone of animal behavior—found that 71% of studies testing for kin recognition in ants did not report the use of blind observation, and, more disconcerting, studies that did not report blind observation reported significantly greater effect sizes than studies that reported blinding (van Wilgenburg and Elgar, 2013). Likewise, herbivory of woody plants was rated nearly 500% higher with unblinded methods compared to blinded methods (Kozlov et al., 2014).
That’s disturbing. And it apparently disturbed Kardish and her colleagues, so they did a survey of nearly 500 papers in journals publishing work on ecology and evoljtionary biology (EEB), seeing if blind studies were used when it was possible to do so. The sad results (my emphasis):
- The authors surveyed 492 recent articles in 13 journals publishing EEB papers, including nine speciality journals and four general ones: Science, Nature, Proc. Nat. Acad. Sci. USA, Evolution, The American Naturalist, Animal Behavior, Proceedings of the Royal Society, etc. Articles were selected before they were examined.
- For every selected study, the authors judged whether or not the results could in principle have been affected by observer bias, and then whether its authors reported blinding methods in the “materials and methods” section.
- Of the 492 articles selected, 248 had results that could have been affected by observer bias.
- Of those 492, a pathetic 13.3% (33 articles) actually stated that they used blind observations. (Of course it’s possible that some studies didn’t report blind observations that were made, but most authors would mention that, at it’s a plus.
- Finally, the impact factor of the journal had no effect on whether or not blinding was used.
For the word-adverse, the authors also provide this superfluous figure (I’m not sure why, but I put it in for grins):
The upshot? Researchers and journals have a ways to go. I agree with Kardish et al. that researchers must report whether or not observations were blinded when possible, and that reviewers and editors demand that information. If studies could have been blinded but weren’t, those should be taken with a grain of salt. Remember, observer bias and confirmation bias are temptations for all of us, and for the good of science we need to institute procedures to prevent them.
I need hardly add that confirmation bias in religion works exactly the opposite way as in science: it’s actually encouraged. Miracles are not to be doubted or investigated carefully, the truth claims of religions other than yours aren’t investigated too carefully, but are dismissed outright, and believers are always looking hard for evidence to support the tenets of their faith.
_______
Kardish, M. R., et al. (2015). Blind trust in unblinded observation in Ecology, Evolution and Behavior. Frontiers in Ecology and Evolution 3, http://dx.doi.org/10.3389/fevo.2015.00051
Coyne, J. and B. Charlesworth (1997). Genetics of a pheromonal difference affecting sexual isolation between Drosophila mauritiana and D. sechellia. Genetics 145: 1015-1030.
Coyne, J., et al. (1994). Genetics of a pheromonal difference contributing to reproductive isolation in Drosophila. Science 265: 1461-1464.
.
