After six months of languishing at Science Online Express, the “Arsenic Paper“, by Felisa Wolf-Simon et al. (full reference below), has finally appeared online. It also appears with eight “technical comments”—criticisms of the methodology and results—a response by the authors, and a note by editor Bruce Alberts (all of these are at the link, though I think there’s a paywall for the original paper).
As you remember, this paper attracted huge attention because of its claim that a bacterium growing in Mono Lake, California was able to incorporate arsenic instead of phosphorus into its DNA and proteins. This in turn led to all sorts of speculations about whether life could have evolved in places (or on planets) that had different arrays of chemicals, or even more than once on Earth. There was a press conference and a lot of hoo-haa promulgated by the authors, including a press conference and a TED talk and Glamour Magazine interview with Wolf-Simon. At the same time, a lot of critics began to weigh in with serious questions about the paper’s results, with some researchers declaring it fatally flawed. Bloggers, scientists, and journalists, such as the Philadelphia Inquirer’s Faye Flam, all noted problems with the paper. And I don’t recall anybody other than the authors defending it.
Summarizing the brouhaha around the paper, Carl Zimmer wrote a Slate piece titled “This paper should not have been published.” He was right.
It’s extremely unusual for a paper to be delayed this long, and then to appear simultaneously with its critiques. Usually the paper is published and then the critiques (with a response by the authors of the original paper) appear in the journal a few months later. Alberts’s note doesn’t do much to clarify the situation:
The Research Article “A bacterium that can grow by using arsenic instead of phosphorus” by F. Wolfe-Simon et al. [p. 1163, (1)] was the subject of extensive discussion and criticism following its online publication. Science received a wide range of correspondence that raised specific concerns about the Research Article’s methods and interpretations. Eight Technical Comments that represent the main concerns, as well as a Technical Response by Wolfe-Simon et al., are published online with this issue . . .
. . . We hope that publication of this collection will allow readers to better assess the Research Article’s original claims and the criticisms of them. Our procedures for Technical Comments and Responses are such that the original authors are given the last word, and we recognize that some issues remain unresolved. However, the discussion published today is only a step in a much longer process. Wolfe-Simon et al. are making bacterial strain GFAJ-1 available for others (2) to test their hypotheses in the usual way that science progresses.
But what happened is pretty obvious. After the blow-up around the paper’s release, Science realized that they screwed up. Their way of making amends was to delay the paper—after all, they couldn’t reject it after it had already been accepted—and publish it simultaneously with the critiques. Having read them, and the authors’ response, you can easily see that the paper wasn’t properly reviewed in the first place. Too many questions remain about the technique, the statistics, and the results. Even I, a biologist far removed from biogeochemistry and microbiology, can see that the paper wasn’t thoroughly vetted.
Over at Slate, Zimmer weighed in again on Friday with an analysis of the scientific sociology of this paper. The big lesson he draws is that we’re entering a new era of science, in which papers can be made or destroyed by post-publication review by blogging scientists:
Redfield [Rosie Redfield, a vociferous critic of the paper] and her colleagues are starting to carry out a new way of doing science, known as post-publication peer review. Rather than leaving the evaluation of new studies to a few anonymous scientists, researchers now debate the merit of papers after they have been published. The collective decision they come to stays open to revision.
Post-publication peer review—and open science in general—is attracting a growing number of followers in the scientific community. But some critics have argued that it’s been more successful in theory than in practice. The #arseniclife affair is one of the first cases in which the scientific community openly vetted a high-profile paper, and influenced how the public at large thought about it.
I think post-publication review is all to the good. Let a thousand voices blossom—even if some of them are misguided. In the end, it’s science, and the truth will out. In the meantime, immediate online criticism is the only way to counteract the hype and p.r. that often attend the release of hot new papers—hype intensified by clueless journalists with no training in science.
The whole thing reminds me of the Darwinius affair, in which a primate fossil was touted as a missing link between the two major groups of primates. There was a lot of buzz and an authorial press conference. Science bloggers raised serious questions and, sure enough, more careful research showed that the vaunted fossil wasn’t a “link” at all. In this era of tight money and lots of scientists competing for attention, the ability to call immediate online attention to problems with high-profile research can only be a boon.
You should begin to get worried about a paper when its authors (Wolf-Simon here, in a video for Dutch television) characterize their results, before publication, like this: “If my hypothesis proves to be true, it will fundamentally alter our understanding of being alive.” Even if the arsenic hypothesis did prove to be true—and I suspect it won’t—it will have miniscule effect on our understanding of being alive. Sheesh!
Wolf-Simon, F. et al. 2011. A bacterium that can grow by using arsenic instead of phosphorus. Science 6034:1163-1166.