In an article in today’s New York Times, “A sharp rise in retractions prompts calls for reform,” Carl Zimmer documents and analyzes the sharp increase in the proportion of papers retracted in the scientific literature. The trend is disturbing: here it is from 2000-2009:
The article notes:
In October 2011, for example, the journal Nature reported that published retractions had increased tenfold over the past decade, while the number of published papers had increased by just 44 percent. In 2010 The Journal of Medical Ethics published a study finding the new raft of recent retractions was a mix of misconduct and honest scientific mistakes.
Retractions, of course, reflect not just scientific misconduct, but honest mistakes that were caught later. My suspicion, though, is that most of them involve either misconduct or the publication of quick-and-dirty results that aren’t checked carefully enough. Zimmer posits that, in case of simple error, the wider proliferation of journals online makes those errors easier to spot. To me this is insufficient to explain the huge rise shown above, but that’s just a feeling. (Of course, there have been well-documented cases of fraud.) The root cause of fraudulent or quick-and-dirty publication is increasing pressure to publish and get grants, leading to more anxiety and careerism:
But other forces are more pernicious. To survive professionally, scientists feel the need to publish as many papers as possible, and to get them into high-profile journals. And sometimes they cut corners or even commit misconduct to get there.
To measure this claim, Dr. Fang and Dr. Casadevall looked at the rate of retractions in 17 journals from 2001 to 2010 and compared it with the journals’ “impact factor,” a score based on how often their papers are cited by scientists. The higher a journal’s impact factor, the two editors found, the higher its retraction rate.
The highest “retraction index” in the study went to one of the world’s leading medical journals, The New England Journal of Medicine [NEJM]. In a statement for this article, it questioned the study’s methodology, noting that it considered only papers with abstracts, which are included in a small fraction of studies published in each issue. “Because our denominator was low, the index was high,” the statement said.
The NEJM’s claim makes no sense to me: the size of the denominator (number of papers with abstracts) shouldn’t affect the average number of retractions, only the variance around the mean.
And let me reproduce what Zimmer says about the changing nature of science, a pernicious trend with which I agree 100%:
In such an environment, a high-profile paper can mean the difference between a career in science or leaving the field. “It’s becoming the price of admission,” Dr. Fang said.
The scramble isn’t over once young scientists get a job. “Everyone feels nervous even when they’re successful,” he continued. “They ask, ‘Will this be the beginning of the decline?’ ”
University laboratories count on a steady stream of grants from the government and other sources. The National Institutes of Health accepts a much lower percentage of grant applications today than in earlier decades. [JAC: The same is true at the National Science Foundation, where funding has dropped to about 7% of applications.] At the same time, many universities expect scientists to draw an increasing part of their salaries from grants, and these pressures have influenced how scientists are promoted.
“What people do is they count papers, and they look at the prestige of the journal in which the research is published, and they see how many grant dollars scientists have, and if they don’t have funding, they don’t get promoted,” Dr. Fang said. “It’s not about the quality of the research.”
Dr. Ness likens scientists today to small-business owners, rather than people trying to satisfy their curiosity about how the world works. “You’re marketing and selling to other scientists,” she said. “To the degree you can market and sell your products better, you’re creating the revenue stream to fund your enterprise.”
Indeed. Getting a grant used to be a means to an end: it provided the money to help us do the research to understand nature. There’s now been a curious inversion of priorities, in which the research itself becomes the means to procuring the end: the grant money, needed by universities to pay for their facilities and faculty salaries. All of us at research universities are feeling increasing pressure to get grants. Once, for example, I was offered a job at another university. My own school offered me a raise and some research money to stay, but in return I was asked apply for more grants—this despite the fact that in my entire career I have been funded by a single, multiply-renewed NIH grant that was ample for my research. I didn’t need any more grant money, but my university did!
Zimmer interviews some scientists who suggest solutions (one of which I heartily approve is to judge the quality of work produced by a scientist more than the grant money accrued or the number of publications), including capping the amount of money that can go to a single laboratory. Will these changes be made? I doubt it. The money culture of science seems unstoppable.
Maybe we should wonder about role models…what might be the single most famous biological discovery of the 20th century, and how did its discover(s) behave?
Endosymbiosis – Lynn Margulis.
Pretty much made her nuts.
As far as I’m aware she didn’t as much ‘discover’ the phenomenon as popularising it. I’m very much an advocate where credit being given where due, so see Schimper (I think 1868), or Mereschowsky (1905). The latter’s paper is very similar to Marguilis’, especially in suggesting *every* organelle must represent a separate endosymbiont, since one/two so clearly did! However, I do agree on the crazy part…
I’m digressing (so please accept my apologies) but fixation on a pet theory seems quite common. I was reading about cures for ageing in the early 20th century the other day and came across Browne-Sequard who helped demonstrate the importance of the adrenal gland and its connections to hormones. And, following such an insight, Browne-Sequard went on to hypothesise hormones were responsible for ageing, especially in males. The ‘monkey gland’ operations became quite popular, as animal testicles were grafted into people to fend of the male ageing process.
Just serves as a word of warning that even if a dicovery is important, and sometimes bringing about the beginning of new paradigm, it doesn’t mean the world revolves around the discovery–which seems to be the fallacy. Perhaps, as Zimmer mentioned, becauses stasis very possibly may represent the beginning of the end.
The system is working. This is likely an artifact of social media and the increased transparency of everything.
Great trend. We are seeing the same in all areas of information.
Nothing wrong with retractions. When was the last time you saw a retraction in religion? I guess in 1992 when the pope finally apologized and conceded the earth moves nearly 400 years after persecuting those who claimed it.
Unfortunately, this is true…
“Dr. Ness likens scientists today to small-business owners, rather than people trying to satisfy their curiosity about how the world works. “You’re marketing and selling to other scientists,” she said. “To the degree you can market and sell your products better, you’re creating the revenue stream to fund your enterprise.”
I see this often in my dental profession, in order to get “publicity”. Especially in “throw-away” journals, which comprise mostly of “case-studies”, just to get their name out. They also do this due to the simplicity of publishing in these journals.
We’ve found also during our Current-Lit Sessions, some flaws in statistics while evaluating more high-end journals.
I’ve found that the most important thing is for the doctor reading the article to be as skeptical as possible and to not take anything lightly… even if it comes from the best journal. Good information can be found in “good” and “bad” journals alike. In my opinion, the more and more the scientific profession grows, the more and more crappy articles we have to sift through to see the good ones.
What’s even more alarming is that studies offering a more sober approach to big papers often are published in obscure journals. Although I suppose it’s less of a problem now when most journals have digital archives…
And that explains why scientists may be ever more vulnernable to tainted grants from the Templeton Foundation.
I’m not sure if this is mentioned as I’m doing this quickly, but we shoul dbe examining the percentage of total publications retracted, not totals. Has the ratio changed significantly?
“…retractions had increased tenfold over the past decade, while the number of published papers had increased by just 44 percent.”
I’m curious what that leaves the ratio at now though.
It is interesting to compare this to the incidence of fraud in theoretical papers in evolutionary biology. It is essentially 0%. Not because theoreticians are paragons of virtue, but because any such paper has to present its argument step-by-step in a way that everyone can follow.
Of course mistakes are possible — I once had a whole paper with a very clever and helpful method that was based on a certain function being the density function of a distribution. It turned out that the function had integral infinity, which dashed that hope. Oops.
But I cannot think of a single case of outright fraud (if we exclude the boring case of someone publishing the same valid result twice, which is not unknown).
How can you perpetrate fraud in a theoretical paper? A lot of theoretical papers in physics prove to be wrong. Fraud only exists if you fudge data, for example by using photoshop to manipulate micrographs (this happened), or claim measurements that are fiction.
The manipulated micrograph I refer to was published in the NEJM.
http://www.nejm.org/doi/full/10.1056/NEJM200207113470213
The authors of the retracted paper hired a lawyer who sued the NEJM (in a Rome court), trying to force them to retract the retraction they had published. These people invited me to meet them and their lawyer in Rome, and I had to listen for an hour to the lawyer explaining that the similarity between the two micrographs in question was pure coincidence. Anyone looking at the micrographs sees right away that this coincidence is highly improbable. What is ridiculous is that this was fought out in court by people that don’t understand statistics. Of course the authors hoped that they would succeed because of a judge who would be impressed by their lawyer (who, incidently, had a revolver lying next to his telephone). The NEJM retraction is still there, so obviously the micrograph manipulators did not succeed.
By editing the outputs of computer simulations.
I also don’t know of a single case of *exposed* fraud in theoretical evolutionary biology. But theoretically it’s not impossible, especially with papers based on (Monte Carlo) simulation models. One could cherry-pick regions of parameter space that confirm a favored (verbal) hypothesis, the ‘representative examples’, while leaving out the ‘inconvenient’ ones. It would be nice if journals oblige authors of such papers to publish the simulation code.
In astrophysics you have people who say that dark matter is the solution to explain some of the strange behavior of certain galaxies (they rotate much faster around their centers than prescribed by their visible mass). Others will invoke a deviation of Newton´s inverse square law, saying that over long distances gravitational forces become stronger than predicted by Newton’s law (MOND or modyfied Newtonian Dynamics). Of course people of both persuasions pick examples that work for them, but you cannot call this “fraud.” You have to view this as a discussion that should stimulate more pointed research and observation.
It is true that when methods of data analysis are presented, that the computer simulations presented to validate them are highly likely to show that the new method is definitely better. And later simulations are much less likely to uphold this.
One hopes that the explanation involves unconscious cherry-picking, not deliberate misrepresentation.
I remember hearing of a case where some people were doing computer simulations of competing methods of data analysis, and a Famous Person who had a Well-Known Method wrote to complain that they were using unrealistic parameter values for their simulations. They had noticed that within the space of possible parameter values, one tiny region up the middle had values on which the Well-Known Method did best. They asked the Famous Person which values they ought to use to be realistic. And the Well-Known Person gave them three points, all marching up the middle of that tiny sliver of region.
“any such paper has to present its argument step-by-step in a way that everyone can follow.”
LOL! The average evo theory paper is unintelligible to most biologists.
Capitalism already kicked democracy’s ass. looks like its kicking science ‘s ass now.
no that is not
science’s ass it is kicking but the ass of its own that _employs_ science as means for perpertual growth of capital
democracy and capitalism are identical twins – they ware both unsustainable and feed off each other ignorance: ignorant voters vote for ignorant politicians who have absolutely no incentive in understanding science
in short science goes as-machine-by-itself on the side and unsustainable democracy and capitalism are taking the momentum of evolution out of ignorance to the point at which the whole system will collapse under the burden of unsustainable coverpopulation and overconsumption
it is only a matter of time when fossil fuel civilization collapses and it is not doom and gloom talk: the horizon is evolutionary in nature and no one can exactly pinpoint when it will happen
when scientists realize that they have no other choice bu ‘disallow evolution out of ignorance’ then science will take over as shepherd of human condition and true sustainability would be possible
until then we will hear an oxymoron of ‘sustainable development’, dumbing down of people into “consumers” and corruption of science-for-the sake-of science into science-as-the money making-enterprise (which is not science at all)
I’m afraid thats true…
Certainly this is a result of the decline of religious values in our society especially among scientists? Certainly? /sarcasm
I admit right off the bat, that I have no knowledge about this topic whatsoever, not having worked in this area. But I do hear reports of more college kids cheating on SATs, term papers, tests, resumes’ and on-and-on. Then they get out into the real world and the pressure to produce is even higher. I wonder…
Though I have to admit, it’s seemed pretty damned hard to cheat in the graduate science classes that I did take at university. At that level, you either knew what you’re doing (it seemed to me) or you were out the door. Of course, it was easier for the ones who knew what they were doing to get away with something.
Hopefully this isn’t a continuing trend that ends up giving science a bad name.
Universities and colleges also treat science programs as profit centers by taking “indirect costs” from grants, often as high as 52%. While I understand that science is capital intensive and requires expensive facilities, equipment, and energy, why is such a high University tax on grants acceptible to the funding agencies?
Those rates are formally negotiated: a team of lawyers and accountants representing the university and a team of lawyers and accountants representing the federal government hash it out.
As to why it’s acceptable, I suppose they’ve decided after hearing the university’s arguments that if they want the science to get done, they have to provide the indirect costs that the university needs to support it.
I have no idea whether it really needs to be that high. I suspect accountants have ways to inflate it. I’d love to see how the negotiations play out though.
Yup. The NSF & NIH need to get their excrement together and put some serious limits on indirect costs. It’s blindingly obvious that big chunks of those indirect costs are -not- being used to fund research infrastructure. It’s a big revenue stream, and so far as I can tell there is basically no oversight after it leaves the NSF or NIH.
Worse yet, the increasing pressure on research scientists to obtain grant money has resulted in a steep decline in the jackal population.
The comic you link to is dead on. Just last week I was having a conversation with two colleagues regarding which font is associated with the highest acceptance rate for NSF and NIH grant proposals. According to my colleague, the study has actually been performed, and there is in fact a bias toward a particular font (I can’t recall which it is at the moment).
Oh, it’s not just you bigshots. This crap is trickling down to those of us at Masters-only and even undergraduate-only institutions, even in the second and third tiers. Fundraising is now regarded as the single most important science-faculty function, the one criterion that can trump all others.
I’m pretty bitter about it.
Are you familiar with retraction watch? http://retractionwatch.wordpress.com/
Retraction Watch has sensationalized scientific error and isn’t very good at discerning ulterior motives from honest limitations. It does the job of striking fear into the hearts of publishers, though (I am one. Unfortunately it isn’t particularly well known among those with most at stake: the authors.
If you’ve ever been bitten by RW, or think you have, be aware that there are people using pseudonyms who are actively seeking papers to attack. They typically pursue a target paper b, interviewing disgruntled readers and then taking the charges their interviews produce directly to the publishers along with some not-so-veiled threats. Some people believe they are affiliated with RW, but no evidence so far supports that and it’s hard to see what RW would have to gain.
However, it’s increasingly clear that someone wants to discredit the peer review system, particularly in high profile journals that take the most risks by reaching for the novel. It isn’t hard for an overactive imagination to implicate religion.
To be honest, folk are using their real name on retraction watch. I have been following something concerning a former member of a former institute I was in. The stuff I have read has been well documented. Someone has gone into a great deal of analysis and posted evidence why it is bogus – the same histological section rotated and photoshopped, the same LANE on a phospho ERK blot duplicated, having its intensity modified and pasted into a fake blot etc.
“However, it’s increasingly clear that someone wants to discredit the peer review system, particularly in high profile journals that take the most risks by reaching for the novel.”
Is it a risk if proper procedure is followed? There is a big difference between a mistake and fraud.
However, I have had some real weird reviews of some of my manuscripts. I have had referees that clearly didn’t get the point or understand the techniques used. I even had one report once that left me wondering if I was sent the wrong review.
Personally I’ve never recomended a paper be published withou some form of modification, and many Journal clubs I have attended have shredded “high impact” papers on basic things like methodology. Perhaps publishers should up *their* game
There is also the issue of the referees not doing their job. I’ve had to turn down requests to review papers on a number of occasions because I was not a suitable referee (but apparently not many people will do that – I wonder why, it’s not as if you get paid for the job). On papers that I had reviewed, I’d found that the article was not significantly different from something the authors had already published, the authors make mistaken claims that could have been easily checked, or the authors couldn’t be bothered doing some simple calculations because they’d rather postpone a simple calculation for the next publication. One of my favorites is the use of irrelevant references; I could never understand why anyone would want to fluff up the list of references. I get the impression that I’m one of very few scientists who check the references and the authors’ previous publications.
On the other end, I don’t mind comments from referees such as “this is not clear; more work is needed to convince me of this” even when the ideas were very simple; after all, if the referee who is supposed to know something of the subject is confused, the article is probably missing something. Then there are the times when you get comments which make it pretty obvious that the referee is a moron … but thankfully I don’t get many of them.
It’s unfortunate that scholarly publications are so vulnerable to these outside factors. I’ve noticed news like this getting used to discredit science as a whole more frequently by those unfamiliar with the process. Still, I agree that there is more to gain by increasing awareness of fraudulent/shoddy work than by ignoring it.
I concur, Jerry. Science has morphed from discovering the secrets of teh Universe into big business. Do we want this? ISTR that at least 50% of businesses fail within 5 years.
In grad school I remember the moment when I realized that my advisor was no longer a scientist. He was a small business owner. Virtually every moment he spent at uni was devoted to grant writing, even though he was tenured. I knew then that this type of high-pressure academia was not for me.
I also recall that a multiple-award-winning prof was denied tenure because he did not bring in enough grant money. He was voted best prof at uni 3 times, won numerous professional scholarly awards for his research, and was extremely well-respected in his field. Nevertheless, the uni denied him tenure for lack of grant money. His response was the same as yours, Jerry. He said his work did not require a lot of money. His reputation didn’t matter at all, nor did the quality of his work. It’s all about the money.
And that is a damn shame.
I am working for a degree doing molecular systematics (Jerry calls us “miscreants”); the entire ethos of “publish or perish” has really turned me off to research. I spend more time at my desk working on broader impacts for the NSF than I do dissertation writing. A friend of my advisor was recently awarded an NSF grant…. the first words out of his mouth were how it would help him keep his job.
Maybe I’ll just deliver pizzas for a living, or join our military industrial complex: both would probably pay better and be a bit steadier than a decade of post-docs, followed by pants pissing hoping for tenure (if I’m even lucky enough to get to piss my pants worrying about tenure).
“The root cause of fraudulent or quick-and-dirty publication is increasing pressure to publish and get grants, leading to more anxiety and careerism”
Within weeks of commencing my PhD program I was pressured to publish multiple papers before experiments even commenced on the grounds that “the results of most experiments are easily estimated”.
I refused.
My supervisor went as white as a sheet when he saw the letter requesting me to do so (from, shall we say, a prominent scientist who would be considered beyond question on such matters). I was told to forget I had ever seen it.
That’s -er- … [speechless]
Would that “prominent scientist who would be considered beyond question on such matters” have been on your papers as a co-author or would he have taken no risk in your ship sinking along with such submissions?
Absolute numbers are not best. It would be helpful to have numbers for a class of top journals and normalized to a rate per published.
Concerning the quick-and-dirty publications with subsequent retractions I wonder about punishment. If committees to evaluate proposals just counts the number of publications named by a researcher, no harm there. If retracted papers are continued to be cited as if nothing happened, no harm there either.
Hmm, I’m not so sure that the increase in retractions is necessarily a bad thing, as it may just be an increase in whistleblowing and people honestly owning up to errors.
However, what Jerry says about grants etc is of course dead right and some of the stories in the comments are truly horrifying, but unfortunately not surprising.
Mathematical subjects are very cheap to do as we just need computers and paper, but this now counts against us as it means we need, and hence get, very few grants.
At least those of you in the US can be thankful that you don’t have the RAE/REF (research assessment exercise/research excellence framework) as well, which formally institutionalises at a national level precisely all of this sort of crap.
I just had a second look at the graphic in the post, and it seems as if fraud is only slowly increasing, when compared to scientific mistake (which I think is a perfectly excuseable reason for retraction, although it doesn’t seem to be *that* widely implemented — perhaps because it’s hard to draw the line between quantitative and qualitative mistakes; consider, for example, the Lombardi et al (2009) XMRV in CFS paper, and the Wolfe-Simon arsenic life paper!)
However, the ‘other’ category seems to be the most rapidly increasing (I’m eyeing this, no statisctics involved in this analysis!), but I am very intrigued: what consitutes ‘other’ reason for retraction, that isn’t fraud or mistake?
My hunch would be ethical reasons (like putting patients at risk), irreproducibility that is due to coincidence not fraud or mistake, legal reasons (e.g. the funding company did not allow the publication) – stuff like that.
The ethical and corporate-permission things should stop a paper being published (if they are breaches of previously accepted agreements), but absolutely shouldn’t be cause for retraction of otherwise sound research results. Doesn’t retraction always mean “That was FALSE, sorry”, which is different from “That was WRONG, I shouldn’t have said that”?
Cat, bag. Not sure what the mechanism is for putting it back.
I agree with the general gist of your post, except for one thing. You imply that higher retraction rates in higher impact factor journals is due to people rushing work to publish in those journals. This could be true, but could it not also be that these are the most highly scrutinised journals and therefore mistakes are caught more often?
The desire for recognition is built into our DNA. We all have ego, pride, and the desire to succeed. The rush (and pressure) to publish is fueled by this. In addition, (as Paul Southworth noted), the passion to correct someone’s error may also be fueled by this. Our pride can motivate both good and evil – can be used for selfish gain or to the betterment of others. Yet, it’s there and aside from retractions from refereed scientific journals, there are huge social, economic, and spiritual implications to our internal drives, passions, wants, and needs. I just hope we don’t blow ourselves up because of this.
http://www.washingtonpost.com/business/economy/doubts-about-johns-hopkins-research-have-gone-unanswered-scientist-says/2013/03/11/52822cba-7c84-11e2-82e8-61a46c2cde3d_story.html
Wash Post March 11th..very interesting related story…(Johns Hopkins/NIH/Nature)