One of the onerous parts of being a research scientist is the pressure to publish lots of papers, which ranks second only to getting grants as a prime inducer of stress. To obtain tenure or promotion, young scientists now must publish prolifically. I remember that Mel Green, one of my mentors as a postdoc at The University of California at Davis, told me that in his day (he’s now in his early nineties, and still pushing flies at the bench), a mere one or two quality publications per year was considered a good output.
No longer. Today almost nobody stands a chance of getting tenure at a major research institution without at least four or five papers per year, good or not. Numbers are important. When we were discussing how the administration at the University of Chicago regarded publications at tenure time, my friend Russ Lande told me, “They may count ’em, and they may weigh ’em, but they won’t read ‘em!” Young students are pushed to submit their papers to Science or Nature, a sure recipe for disappointment since those journals are so selective. The pressure also leads to what I see as a destructive level of competition and ambition in young scientists.
My own output has been modest: I have 119 peer-reviewed papers since I started graduate school, which works out to about 3.1 per year. On the other hand, I do have two books, and pride myself on not gratuitously slapping my name on my students’ papers—one reason why some scientists with big labs can have more than 600 publications in their lifetime!
In Monday’s Guardian, David Colquhoun, a well known pharmacology professor at University College London, a fellow of the Royal Society, science critic, and author of the website Improbable Science, decries the publication frenzy that characterizes modern science. His essay, “Publish-or-perish: peer review and the corruption of science,” points out how the the publication culture has spawned a number of low-quality “peer reviewed” journals, some of such absymal standards that they’ll publish anything.
Peer review is the process that decides whether your work gets published in an academic journal. It doesn’t work very well any more, mainly as a result of the enormous number of papers that are being published (an estimated 1.3 million papers in 23,750 journals in 2006). There simply aren’t enough competent people to do the job. The overwhelming effect of the huge (and unpaid) effort that is put into reviewing papers is to maintain a status hierarchy of journals. Any paper, however bad, can now get published in a journal that claims to be peer-reviewed.
Colquhoun gives one example of an infamous paper by a group at Exeter University that studied the medical efficacy of acupuncture. Despite the finding of at best a tiny effect of the procedure (barely a placebo effect!), the authors—and the journal (The British Journal of General Practice)—trumpeted that acupuncture “resulted in improved health status and wellbeing.” Have a look at the figures Colquhoun presents to see how “improved” the status really was! (note that Colquhoun’s column is followed by a response from the study’s authors as well as the journal’s editor, standing by the paper). But clearly, there are substantial problems with the research, and the results are so unimpressive that they’re hardly an endorsement of acupuncture.
Colquhoun’s solution is self publication:
So what can be done about scientific publishing? The only service the publishers provide is to arrange for reviews and to print the journals. And for this they charge an exorbitant fee, a racket George Monbiot rightly calls “pure rentier capitalism”.
There is an alternative: publish your paper yourself on the web and open the comments. This sort of post-publication review would reduce costs enormously, and the results would be open for anyone to read without paying. It would also destroy the hegemony of half a dozen high-status journals. Everyone wants to publish in Nature, because it’s seen as a passport to promotion and funding. The Nature Publishing Group has cashed in by starting dozens of other journals with Nature in the title.
Interestingly, Colquhoun suggests Mel Green’s formula of “an average of two original papers per year” (and adds that scientists should hold only one research grant at a time, a presciption that I agree with).
An obvious problem is that junior people would be afraid, in the comments section, to criticize more established ones. Colquhoun suggests that comments can be anonymous: after all, reviewers’ comments in regular journals are not attached to names. As for worries of fraud because of the lack of peer review, Colquhoun responds:
Deer [Brian Deer, a journalist who exposed Andrew Wakefield’s vaccine/autism fraut] has recently backed a proposal from the House of Commons Science and Technology select committee that an official regulator should be appointed to police science. I don’t think this could work. Is the regulator going to repeat experiments, or even check original data, to make sure all is well? In all probability, a regulator would soon degenerate into yet another box-ticking quango, and end up, like the Quality Assurance Agency, doing more harm than good. The way to improve honesty is to remove official incentives to dishonesty.
By and large, the problem does not arise from outright fraud, which is rare. It arises from official pressure to publish when you have nothing to say.
The last sentence is absolutely on the mark.
Colquhoun’s proposal has substantial merit. What I see as the biggest problem is that it forces tenure and promotion committees, as well as granting agencies. to actually read and evaluate a candidate’s papers, rather than relying on secondary indicators of “quality” like journal “impact factors,” grant dollars, or numbers of publications. The main impediment to this system is the laziness of university administrations.
h/t: Dom
Why isn’t all of this online anyway? Ego?
Physicists and astronomers publish online in arXiv. It has some kind of peer review whereby bad papers disappear after a while
No, papers on the arXiv do not “disappear after a while”. In fact it is impossible as far as I know, to completely delete a paper from the arXiv. Records of retracted papers are also maintained indefinitely (of course, with a clear marker that the paper was retracted).
The ultimate model of academic research and discovery will probably take a form none of us can anticipate today, but in the meantime, perhaps some sort of accreditation standard might be applied to journals, so that authors have two numbers to report: Both total publications, and total publications in accredited journals. Getting accredited would of course require certain standards of peer-review. Good idea? Bad idea? Already being done? Looking for some peer review here.
I doubt you meant to say ‘fraut.’
http://www.urbandictionary.com/define.php?term=fraut
That entry is just someone trying to be hip by shortening and misspelling “frottage”.
Perhaps “a” and “u” cost less than “o”. Perhaps he suffered a failed “o” harvest.
How to prevent a sock puppet population explosion ?
Both sock puppets and anonymous postings by total braindeads are a problem with open publication on the web. Do we really want creationist comment in large quantities to every evolutionary biology paper?
There are three problems:
1 the publisher rackets
2 university administration
3 the web itself
The publisher racket might be undercut if all national science foundations insist on open-access publishing in non-commercial papers.
Open publication on the web still means specialized websites with agreed names. Moreover, someone has to service those websites ad infinitum!, or much of science will disappear. Websites are much more ephemeral than paper.
The university administrations might reverse policy, and insist on quality as assessed by the field. That is going to be difficult, as long as university administrations are bureaucrats rather than scientists of the field.
Another problem is this: is there an overproduction of PhD students? Or is the money available below par? Science is a disastrous career decision nowadays, with far too many very competent people pursuing far too few grants. To insist on fewer PhD students might be another possibility – but will reflect badly on many labs ….
A semi answer for specialised websites and misuse of anonymity could be for the universities to take it up either individually or as groups.
Means there is a lower risk of sites disappearing over the years.
Also a more rigorous, and independent, registration process could be used than would be available for publishing on the authors own sites.
Be problematic to get right but would allow for the user to be identified before getting commenting rights but being able to remain anonymous to the other users.
While it wouldnt be cheap to do properly with their existing IT requirements the universities already have a fair amount of the infrastructure in place.
I think papers as blog (yes “blog”) posts is not the way to go. One simple (simplistic?) way to solve the problem would simply be to improve the quality of review with a concomitant decrease in the number of publications. Of course the implementation of this is a challenge. I suggest that quality reviewing be incentivized through both tenure and grants.
*clenched-tentacle salute*
If you look back at the experimental and descriptive developmental biology literature of the mid-1800s through the 1930s, many of the publications were monographs or long papers that covered a whole area and multiple experiments. In fact, Darwin’s habit of publishing as complete a story as possible was not unusual. Of course, scientists then actually knew how to write!
The only downside of the lengthy publications was competing researchers were quite outspoken about their views of the competition, often coming close to slander. But the slanging was at times enjoyable to read.
Today, we get the LPU (least publishable unit) as the subject of a paper, to pad the number of papers published each year, and everything written in the passive tense. Bleh!
I couldn’t possibly agree more with the points in this posting. Thank you, Jerry! I’ve been a research mathematician for 30 years, publishing about 1 or 2 articles per year (no “fluff”), and although I somehow got tenure back in the day, I am now considered an unproductive member of my department at “merit review” time. It is amazing how easy it is to publish borderline nonsense in some math journals. Clever politial types set up journals where the same people referee and publish each other’s work over and over.
The situation in biology (at least in mathematical biology) is no better as far as I can see, and in some ways it seems to be worse. In mathematics a good paper always gets published. In biology, papers can get rejected, without even being read, because they are unorthodox. I’ve learned this the hard way.
I think it won’t be long before papers are self published. I never go to the journals to find papers (except old ones) since pretty much everybody has their work available on their websites. If there was a standard way to comment on all these papers the best work would quickly float to the top, whether it appeared in a journal or not.
“By and large, the problem does not arise from outright fraud, which is rare. It arises from official pressure to publish when you have nothing to say.”
Are there sufficient controls in place to prevent fraud, especially in highly specialized research where there may be only a few people in the world who understand that research?
No, there are virtually no controls in place to prevent fraud, and that’s what Brian Deer was complaining about in the recently-cited article. Despite this, it seems to be a relatively rare occurrence… so one ought to be wary of this being “a solution in search of a problem.”
Not being in the field (I’m a software developer now) of philosophy of science anymore, it disappoints me I cannot spend the time to test a hypothesis about fraud I and others have had. Namely, the closer one gets to “affecting humans” the more likely it is. Probabilistically, of course. I think one has to control for the maturity of the science, too, which is difficult (and contentious). Any sociologists of science (no pomos!) want to do this? Or maybe it has been done.
What are the various things that make fraud more harmful than honest errors?
Among doctors I am told there are similar problems so there are lots of indifferent case studies published, & inferior trials. A senior surgeon I know was telling me of a colleague who writes articles on subjects in which he is not a specialist by doing reviews. I am no expert but whenever I hear a news story which trumpets a health finding then it turns out it is NOT original research but a review of other studies, I am cautious. Don’t get me wrong – reviews are important, but they have to be really rigorous. Some people who should know better have curious ideas about levels of evidence & sample sizes.
A salute to your stand on not taking credit for your student’s work. As a public health dentist working in a teaching and research institute I have seen the plight of academicians and researchers in this regard and especially I feel helplessly angry on seeing how post-graduate students are exploited in the field of dentistry in India. This pressure has suddenly produced umpteen number of journals in the past couple of years when these standards were introduced by Universities and has lead many such journals making lot of money!
…or you could toil away for years on a doctoral dissertation in the humanities with virtually no contact or encouragement from anyone other than your advisor, only to find that the ‘big’ journals that you’re expected to publish in have been taken over by “post-modernists” who find “facts” a quaint passé notion at best and couch everything in incomprehensible pseudo-marxist queer theory gibberish, thus rendering your dissertation topic and your entire field completely unpublishable except in a few honest journals that the tenure committee who knows nothing about your field (or else they wouldn’t have hired you to teach it) have never heard of, thus rendering your entire career path even more of a waste of time than your parents predicted years ago, and despite your ability to write run-on sentences your inability to write lengthy paragraphs in post-modernist gibberish will keep you from even being able to present at the ‘big conferences.’
…not that I’m bitter.
Ouch.
…but I’ve still got you beat: Bachelor of Music in Orchestral Trumpet Performance.
I hear there’s a university somewhere in Scotland that offers a BM in bagpipe performance….
Cheers,
b&
I guess that explains the programming gig…
So…I’d be an accountant had I taken up the pipes? I suppose that might be a small favor to be thankful for….
b&
The trumpet does resemble the abacus
It does?
Well, both are effective forms of birth control, so I suppose you might have a point….
b&
It is only a matter of time before some ‘mathematically inclined student’ goes on to some ‘traditional’ professorship of relatively fixed regimen/hours, but some other similarly inclined and capable student opts instead for TRASH-COLLECTING that ‘no one wants to do’ -but of more or less equal pay and fewer labor/hours and more ‘free time for mathematics or other intellectual or life-style-and-quality interests’ -a ‘multistationality of occupation’ therein.
from
The Nature and Course of Human Evolution
as
The Basis of Economic Policy
at http://www.condition.org/econpol.htm
“…pressure to publish when you have nothing to say.” Finally something that relates to both science and religion.
As long as the titles look different enough from each other you can recycle and reuse.
The Deepak Chopra School of publishing when you have nothing to say.
But, lo, you are being closed-minded! For never thy essential potentiacies be enlightenly realisated were mindfulness quotients ne’er Z-Quadrant Elevation corrected post-willing!
The other problem with the post-publication review model is that, like it or not, publication of any sort imparts an air of respect in the eyes of the public, regardless of the quality of the paper or what comes after. Anybody who wanted to believe anything could cite a paper in support of their chosen nonsense du jour, and 95+% of the general population (as well as a shameful fraction of science journalists) will not be able to tell the difference between that and a quality paper… even if other researchers excoriate the paper in the comments, so what? The headline is all people read anyway.
I think post-publication review has a lot of positives in terms of scientists effectively vetting each other’s work, but in terms of public perception, it seems like it stands to make a bad situation worse…
When the civilizational collapse becomes manifest in more places on the planet and finally reaches so called “developed” countries the general public will matter even less in terms of mankind’s survival as the species
Then scientists will realize that they need to cooperate more with other scientists in order to produce and institutionalize policies that relevant to the survival of whatever diminished human population.
tBy that time those scientists will be part of “ruling classes” and the so-called middle class will long be extinct.
The current situation with publishing of material in scientific journals that does not advance science is the reflection of general overpopulation and the fact that it is not physically possible to meritably employ all the people.
The more people we have the less meritable emplooyment per capita the more noise produced to be passed for genuine scientific output.
The original article gets abstracted but never the letters that get published later!
Unfortunately, academia has sold its soul for 30 pieces of silver. Grants should never be a requirement for promotion and tenure. A grant is its own reward, and does not require that it also be part of the tenure process.
The current system biases research toward problems that can quickly generate papers, even if not important. It biases against research on hard problems with no guaranteed results.
+1. Despite all of the lip service by the NSF and NIH about “risky” and “transformative” research, the PI’s with 3 R01’s already who work in the same well-worn groove seem to only get more. Not that I’m bitter or anything.
Worse, it invades fields which should be working on large problems (philosophy, for example) and have great deal of difficult with unification. Fortunately, there are some steps (with the joining logics movement, or whatever it is) and people like Bunge and Armstrong who refuse to do what the former calls “sawdust philosophy” but those two are far from young …
can someone please tell me what the term “still pushing flies at the bench” means?
Google comes up emptyhanded.
thx in advance
At a guess, using flies in (bench) experiments.
The inability of google to learn and express area dependent/expert language is abysmal.
For example we all know that “Time flies while you’re having fun” means to use a stop watch to measure the speed of flies as you experience enjoyable activities. English is so precise.
More specifically, moving fruit flies (drosophila) in and out of milk bottles. These flies reproduce fast, so you can study several generations in a short time (and their chromosomes are very visible in the microscope). Unfortunately, glass milk bottles are becoming a rarity, and genetics labs, who need large quantities of these bottles, have to resort to expensive lab glassware that just looks like …. milk bottles. But here in Germany we are infestated with fruit flies and they seem to like wine bottles, especially open ones, so I would suggest that labs switch to wine bottles.
You know, I think I might know of a certain evolutionary biologist who studies fruit flies who might know where to find a supply of empty wine bottles.
Does glass color matter?
Cheers,
b&
Ah well, you might get less UV induced mutations in Bordeaux bottles than in Muscadet bottles.
I thought silica glass was generally opaque to UV, especially UVB and shorter? Is UVA mutagenic?
b&
Yes, normal glass blocs a lot (but not all) of the UV spectrum. I think we need to look at a study of the UV absorption properties of wine bottles before inviting fruit flies to join the experiments. In fact, wine bottles are designed to stop UV degrading wine.
Alas, my spectrophotometer isn’t designed to reach into the UV. I would, however, be delighted to assist…were it not for the fact that UV-sensitive spectrophotometers cost (as I understand) well into five figures and up (way up).
It’s a shame, too — creating printer profiles for papers with fluorescent whitening agents (and, ideally, even fluorescent inks) to be displayed under lights with different UV content would actually be practical otherwise. And, believe me, there’re all sorts of really neat things you could do with controlled fluorescence.
But maybe Jerry has some connections at the university who’d be willing to trade access to such an instrument in exchange for…ah…samples of the bottle’s contents?
Cheers,
b&
If you want glass bottles to keep flies in, Starbucks Frappucino bottles are perfect. They can be plugged with large cotton balls.
As far as the flies themselves, technically speaking D. melanogaster is a vinegar fly (it eats the yeast that grows on rotting fruit, rather than the fruit itself) instead of a fruit fly. Red wine vinegar is one of the best baits for traps.
The term “fly pushing” is the generic term for what drosophilists do.
http://www.amazon.com/Fly-Pushing-Practice-Drosophila-Genetics/dp/0879694920/ref=sr_1_1?ie=UTF8&qid=1315443694&sr=8-1
I got to know Mel Green in graduate school at UC Davis, and he is quite a character. He was, I believe, the first person to ever demonstrate that recombination could happen within a gene (at the vermillion locus).
I hadn’t heard of “pushing flies” prior to this article; but, back when, there was a journal called Worm Runner’s Digest – which later changed its name to Journal of Biological Psychology. See http://en.wikipedia.org/wiki/Worm_Runner's_Digest for more information.
Putting some sort of business-type metric on scientific work surely does not help this sort of mentality either.
Another problem not mentioned so far is researchers publishing what is essentially the same paper in 3 or 4 different journals.
There are a few online publications where readers are encouraged to submit comments during the review period. Generally there is no anonymity nor do I and my colleagues see a point to anonymity. The point of the exercise is to defend your work against valid criticism. I wouldn’t recommend self-publication; I think it is better to have something organized so that publications are still grouped by fields of interest. This is a practical consideration since it’s much easier to go through a list to find and critique items of interest.
Self-publishing is the desperate last ditch choice of a ummm uhhh mad scientist
That is not going to happen, and you have said why yourself. As long as not everybody boycotts all scientific journals at the same time, self-publishers will not be hired, will not be promoted and will receive no grants, and that is the end of that.
I must also say that I do not have such problems with the system of publishing in journals per se. I know where to look for the relevant research – only a handful of decent journals instead of 20,000 blogs of very varying quality. And publications are vetted and consistently layout-ed before publication.
This is not only about peer review. I shudder to think how readable “publications” of some colleagues would be if they could just plop them onto the internet. Common problems I run across in peer review: inconsistent formatting; grammar; typos galore; incoherent sentences, especially if the author is not a native speaker of English; references missing in the literature list; rambling and pointless parts in the introduction and discussion. It is a good thing that all this is corrected before publication instead of people having to prod a potentially unresponsive author in a comment thread.
No, there really are only two problems with the system as it currently works, and they can be addressed without self-publication:
– The fetishization of impact factors and quantity instead of relevance and quality; this could be solved by selection committees becoming more sensible about it, which while unrealistic is much more realistic than selection committees accepting what are effectively blog posts as publications.
– The journal racket; this could be solved by everybody moving to online only open access journals, a process that is taking place although slowly. It will get momentum to the degree that these open access journals achieve higher citation rates and impact factors.
The poor schmoe who corrects the problems you outlined above deserves to earn a few pennies for their trouble. I don’t see how that can be accomplished in an open access environment.
The poor schmoe who corrects the manuscript should be the author themselves – after a reviewer like myself has told them “your manuscript cannot be published like that”. I am fully willing to peer-review for free while other colleagues review my papers for me. It is a service to the scientific community.
Thanks for this nice write-up. I’m certainly not the first person to have suggested that one or two papers a year is all that can be expected. Not being a fly person, I didn’t know about Mel Green. Thanks for the information.
Of course I’m not really sure that post-publication review would work. It hasn’t been tried seriously yet. But I do know that something has to be done about the racket being operated by academic publishers, and, still more important. something has to be done about bean-counting selection committees. They not only make science a rather unattractive career, but they risk making it less honest than it should be.
As an IT person I see the “self publishing” concept as a social networking solution needed with a particular workflow and identity management.
I can envisage something more akin to a wiki than a blog (to allow comments in context from reviewers), with formatting and meta data imposed for consistency, with articles having various states they go through; from “open for comment”/”under final review”/”peer reviewed” or similar and with metrics for quality of reviewer as well as the article based on feedback on ones feedback (eg people who review well rise to top and their future comments get more prominence while poor reviewers have their comments hidden by default or similar)
Such a system needs to be unified and not “self” hosted, it needs multiple official off-line hard copies to be made of works which pass muster (to try to ensure new knowledge is preserved beyond the lifetime of the system)
It would need institutions to reward good reviewers as well as good publishers to encourage that activity. They could also be publically rewarded for their contributions in a similar way to Nobel prizes perhaps.
It needs some form of ID management where individuals can contribute annonomously if they choose, but can also be validated. This would add weight to a metric as a good reviewer.
It would need the press to understand the system and the processes and report on articles accordingly and responsibly (!)
And probably needs to all be hosted/run by an international government agency.
Etc.
Now all I need is a research grant to investigate this further and a journal prepared to let me publish the research in…:)