After six months of languishing at Science Online Express, the “Arsenic Paper“, by Felisa Wolf-Simon et al. (full reference below), has finally appeared online. It also appears with eight “technical comments”—criticisms of the methodology and results—a response by the authors, and a note by editor Bruce Alberts (all of these are at the link, though I think there’s a paywall for the original paper).
As you remember, this paper attracted huge attention because of its claim that a bacterium growing in Mono Lake, California was able to incorporate arsenic instead of phosphorus into its DNA and proteins. This in turn led to all sorts of speculations about whether life could have evolved in places (or on planets) that had different arrays of chemicals, or even more than once on Earth. There was a press conference and a lot of hoo-haa promulgated by the authors, including a press conference and a TED talk and Glamour Magazine interview with Wolf-Simon. At the same time, a lot of critics began to weigh in with serious questions about the paper’s results, with some researchers declaring it fatally flawed. Bloggers, scientists, and journalists, such as the Philadelphia Inquirer’s Faye Flam, all noted problems with the paper. And I don’t recall anybody other than the authors defending it.
Summarizing the brouhaha around the paper, Carl Zimmer wrote a Slate piece titled “This paper should not have been published.” He was right.
It’s extremely unusual for a paper to be delayed this long, and then to appear simultaneously with its critiques. Usually the paper is published and then the critiques (with a response by the authors of the original paper) appear in the journal a few months later. Alberts’s note doesn’t do much to clarify the situation:
The Research Article “A bacterium that can grow by using arsenic instead of phosphorus” by F. Wolfe-Simon et al. [p. 1163, (1)] was the subject of extensive discussion and criticism following its online publication. Science received a wide range of correspondence that raised specific concerns about the Research Article’s methods and interpretations. Eight Technical Comments that represent the main concerns, as well as a Technical Response by Wolfe-Simon et al., are published online with this issue . . .
. . . We hope that publication of this collection will allow readers to better assess the Research Article’s original claims and the criticisms of them. Our procedures for Technical Comments and Responses are such that the original authors are given the last word, and we recognize that some issues remain unresolved. However, the discussion published today is only a step in a much longer process. Wolfe-Simon et al. are making bacterial strain GFAJ-1 available for others (2) to test their hypotheses in the usual way that science progresses.
But what happened is pretty obvious. After the blow-up around the paper’s release, Science realized that they screwed up. Their way of making amends was to delay the paper—after all, they couldn’t reject it after it had already been accepted—and publish it simultaneously with the critiques. Having read them, and the authors’ response, you can easily see that the paper wasn’t properly reviewed in the first place. Too many questions remain about the technique, the statistics, and the results. Even I, a biologist far removed from biogeochemistry and microbiology, can see that the paper wasn’t thoroughly vetted.
Over at Slate, Zimmer weighed in again on Friday with an analysis of the scientific sociology of this paper. The big lesson he draws is that we’re entering a new era of science, in which papers can be made or destroyed by post-publication review by blogging scientists:
Redfield [Rosie Redfield, a vociferous critic of the paper] and her colleagues are starting to carry out a new way of doing science, known as post-publication peer review. Rather than leaving the evaluation of new studies to a few anonymous scientists, researchers now debate the merit of papers after they have been published. The collective decision they come to stays open to revision.
Post-publication peer review—and open science in general—is attracting a growing number of followers in the scientific community. But some critics have argued that it’s been more successful in theory than in practice. The #arseniclife affair is one of the first cases in which the scientific community openly vetted a high-profile paper, and influenced how the public at large thought about it.
I think post-publication review is all to the good. Let a thousand voices blossom—even if some of them are misguided. In the end, it’s science, and the truth will out. In the meantime, immediate online criticism is the only way to counteract the hype and p.r. that often attend the release of hot new papers—hype intensified by clueless journalists with no training in science.
The whole thing reminds me of the Darwinius affair, in which a primate fossil was touted as a missing link between the two major groups of primates. There was a lot of buzz and an authorial press conference. Science bloggers raised serious questions and, sure enough, more careful research showed that the vaunted fossil wasn’t a “link” at all. In this era of tight money and lots of scientists competing for attention, the ability to call immediate online attention to problems with high-profile research can only be a boon.
You should begin to get worried about a paper when its authors (Wolf-Simon here, in a video for Dutch television) characterize their results, before publication, like this: “If my hypothesis proves to be true, it will fundamentally alter our understanding of being alive.” Even if the arsenic hypothesis did prove to be true—and I suspect it won’t—it will have miniscule effect on our understanding of being alive. Sheesh!
_____________
Wolf-Simon, F. et al. 2011. A bacterium that can grow by using arsenic instead of phosphorus. Science 6034:1163-1166.
This certainly is a problem. Even a depressing number of “science journalists” all to often make the kinds of “misstatements” that somebody who managed to not fall asleep in junior high intro-to-science classes shouldn’t make.
Ms. Flam seems to be in the minority of good science journalists. She’s an engaging writer, and I get the impression that she’d have been a successful researcher had she chosen that path instead of journalism. At the least, she seems to “get it” more often than not.
Mom has a subscription to Science News, and they seem to “get it” as well. It’s written for the general public, but they pull from primary sources rather than press releases. They also seem to have some sort of internal review process; I seem to recall (sorry, no specifics) them expressing appropriate skepticism for some of these recent bad science media sensations .
In the video, Dr. Wolf-Simon goes on and on about the toxicity of Mono Lake. I’ve swum in the lake; it was a popular destination for camping trips when I was growing up, sort of like California’s answer to the Dead Sea. Maybe it’s not the smartest place to go for a dip, but it’s certainly not the instantly-lethal toxic waste dump she makes it out to be. (Or, at least, it wasn’t a few decades ago.)
Cheers,
b&
The whole press relese thing as a way of breaking a big science story is, I suppose not new, but seems increasingly important with one eye on sponsors & gerenating research funds. The problem is there for unglamourous science – it will be neglected.
I’m guessing, because I don’t know otherwise, that, over these six months, the authors have done nothing experimentally to address criticism of their original research. If so, that’s telling.
Yes, I think you’re right, and that seems to be a huge problem. It’s almost as if they don’t WANT to do the right studies. Fortunately, somebody will, as they’re having to make the bacterium available to other researchers.
In his Slate piece, Carl Zimmer makes the good point that the online posting of the original paper last December was not quite publishing it; the version published Friday in the journal differs somewhat from what was posted. The critiques of the arsenic work, though, have not in fact been published in Science, only links to online postings of them have been; readers of the journal Science will not be able to read the critiques.
The notion that “post-publication review” is something new is odd. There’s always been post-publication review: that’s how most scientific criticism is done (the pre-publication review being limited to at most a handful of reviewers and editors). Zimmer cites a few already published criticisms, and the critiques of Redfield et al., had they remained mere blog posts, would have left the status of the work in a murky world of online discussion. Critics, like the original authors, must eventually turn their critiques into publishable arguments. What is new is the speed with which both informal critique and published critique can be made and spread (blogs vs. letters, and the rapidity of publication using electronic communication).
Yes, Greg’s right – though I guess Zimmer would claim that speed and interactivity, which the web does so well and dead trees and print journals did so poorly, will make a qualitative difference.
However, note that this was the whole idea behind PLoS ONE, which has been a massive success and is being imitated by Macmillan (who publish Nature), the Society of Biologists (JExpBiol, Development etc) and others. PLoS ONE said it would publish stuff as long as it was sound (probably wouldn’t apply to #arseniclife), and that the peer review would take place in the comments thread of the PLoS ONE site. Guess how often this happens? As near enough to 0 as not to be significantly different.
The reason is obvious – we don’t have enough time to read the literature, and occasionally be referees, never mind go and start commenting on everything. In other words, unless an article is ludicrous or over-hyped it will be ignored. The bulk of stuff, even if it’s plain wrong, will be ignored. Just like in the old days.
That is a really interesting point. I would argue that it is a lot harder to browse an online journal and – perhaps a generational thing – to sit and read for any length of time from a screen & absorb the information in a less than superficial way.
I worry that with “post-publication review,” it’s going to be harder for lay people to know whether a paper is credible. There is the assumption that if it was published in a reputable journal, then the content is good. When I wrote papers in college, I didn’t usually google my sources to see what other scientists were saying about them. (Mind you, this was 10 years ago, when you needed library resources to find electronic copies of published research, and I wasn’t a science major.)
The literature has always been full of crap. Now you can easily determine it through a quick Google search. If you really want to know whether a paper is worthwhile, you actually have to see if it survives the test of time, which its increasingly easy using online tools like Web of Knowledge or Google Scholar. Just wait a year or two and look at the number and tenor of citations. That will give you a good idea about the quality of the paper.
The literature has always been full of crap.
Oh? exactly how much of it is “crap” do you think? Not been my experience; 90% of the papers in the journals I’ve read over the last 30 years have not had significant errors in methods or results or conclusions. Are you perchance reading popular science magazines instead?
Just wait a year or two and look at the number and tenor of citations. That will give you a good idea about the quality of the paper.
no, it actually won’t.
In fact, the vast majority of papers published rarely get cited by others.
It doesn’t mean they are bad, it just means the specific topic involved wasn’t of great interest to a large number of other scientists.
even where there are multiple papers on an interesting subject, it STILL isn’t the case that citation numbers indicate overall quality.
sometimes, it’s just timing, or exposure, or the specific person who did the work.
Actually, as was pointed out upthread, this is really just the online version of what has gone on in academia for eons.
whenever an interesting paper comes out, it is almost always vetted in various grad student seminars or similar, and each relevant group in each uni comes to a consensus about what it does or does not contribute to their field of interest.
So, if you have a paper that is of interest to you as a layman, one way to get an idea of its relative merit is to ask other scientists in the same field about it.
It has always been this way, and the addition of online access just makes this EASIER, not harder.
so, while you’re right to be concerned about a layperson not necessarily being able to properly vet a journal article, this is not a new thing.
Interest laypersons can always contact the author with their questions, or other scientists.
I’d also note that it is very common for journals to publish letters from scientists in the same field legitimately critiquing a paper they had previously published.
so you’ll see the vetting battle ongoing in the journals themselves on occasion.
“If my hypothesis proves to be true, it will fundamentally alter our understanding of being alive.”
Hey, I got hypotheses like that!! Of course, they mainly involve alcoholic drinks in pineapples with parasols.
I have very little confidence in anything NASA says these days….they don’t even seem to know which way the Earth spins any more 🙂
http://www.nasa.gov/multimedia/videogallery/index.html?media_id=86156471
The link to Friday’s Slate paper is to the second page – here’s a link to the whole thing as a single page: http://www.slate.com/id/2295724/pagenum/all/#p2
I also found the notion that post-publication-peer-review is new a bit strange, as time is supposed to tell whether someone’s theory holds up under further scrutinization. Although I can see that the use of internet, and the speed, changes the game.
But I was a bit shocked by some comments in the news article at Nature (Will you take the ‘arsenic-life’ test, http://www.nature.com/news/2011/110601/full/474019a.html), where researchers basically say that they don’t want to spend their (or rather, their students’ …) time on it, because if you disprove it, where are you going to publish it … ? Maybe I should read that as that it is telling that people are not willing to spend their time on it. But still.
Lovely. And I mean that literally.
It’s quite lovely to see the entire process of science at work. Especially the skepticism and self-correction.
Imagine if some other “different way of knowing” behaved in such a manner?
Well said! +1