There seems to be a bit of science-dissing going on in the U.S. A while back Jonah Lehrer wrote a New Yorker piece about the “decline effect,” whereby positive scientific results tended to erode over time. Then a Greek statistician named John P. A. Ioniddis reported that most results published on medical research were wrong, for they couldn’t be reproduced. Finally, last week science writer George Johnson at the New York Times started his new column, “Raw Data,” reprising the claim that most research is not reproducible, and therefore likely wrong.
The problem is that Johnson’s column, “New truths that only one can see,” deals again with medical research: mainly drug studies. But the implication is that most science is wrong, and Johnson’s column comes off as a claim that new scientific findings really can’t be trusted. He doesn’t talk about all the findings in physics, biochemistry, chemistry, and other fields that are implicitly repeatable because researchers rely on them when doing further work. If the original studies were wrong, we’d know it. DNA will remain a double helix, benzene will never have more than six carbon atoms in its ring, and HIV/AIDS is caused by a virus. We also know the speed of light in a vacuum is invariant, and we know that speed to an accuracy of one meter per second. The Standard Model of particle physics still holds.
I was going to take Johnson’s column apart, but then I heard from science writer Faye Flam, who was in the process of doing so herself. She’s just published her critique at Science Tracker, and you should have a look. The fact is that despite Johnson and Lehrer’s barking headlines, the caravan of science moves on. Physics, for instance, has made enormous progress, and dubious findings like the faster-than-light neutrinos soon get weeded out. Remember cold fusion? If not, why not?
Well, now there’s another faux-kerfuffle. As you may know, Stephen Hawking has revised his description of black holes, arguing that while they still exist, they aren’t bounded by “event horizons,” an edge from which no information or light can escape. But other scientists disagree. Our Official Website Physicist, Sean Carroll, weighed in at National Geographic Daily News:
“I would caution against any belief that Hawking has come up with a dramatic new solution answering all questions regarding black holes,” said theoretical physicist Sean Carroll at the California Institute of Technology, who did not participate in this study. “These problems are very far from being resolved.”
Indeed. And that inspired a hilarious spoof of Michelle Bachmann at The Borowitz Report of the New Yorker:
Dr. Stephen Hawking’s recent statement that the black holes he famously described do not actually exist underscores “the danger inherent in listening to scientists,” Rep. Michele Bachmann (R-Minnesota) said today.
Rep. Bachmann unleashed a blistering attack on Dr. Hawking, who earlier referred to his mistake on black holes as his “biggest blunder.”
“Actually, Dr. Hawking, our biggest blunder as a society was ever listening to people like you,” said Rep. Bachmann. “If black holes don’t exist, then other things you scientists have been trying to foist on us probably don’t either, like climate change and evolution.”
Rep. Bachmann added that all the students who were forced to learn about black holes in college should now sue Dr. Hawking for a full refund. “Fortunately for me, I did not take any science classes in college,” she said.
Borowitz’s column was sent to me by at least one reader who, not realizing that the column is a spoof, thought that this was Bachmann’s real reaction. The sad part is that it’s close enough to reality to be a Poe.
Regardless, it is disheartening to see these science journalists using the irreproducibility of some medical studies—and I agree that many can’t be repeated, as they’re based on limited populations or families—to imply that science itself is deeply flawed. Sure we get things wrong, but if everything was wrong we’d never make any progress! And look how much farther we are in physics, molecular biology, chemistry, and, yes, medical research, than we were thirty years ago. Science seems to have a way of discarding away its mistakes.
I wonder if George Johnson takes the drugs that his doctors prescribe. . .
//
///
Remember cold fusion? If not, why not?
Because they brought so much ridicule onto themselves that they renamed the alleged phenomenon low energy nuclear reactions. When they wear that one out, maybe they’ll try yet another name, like Blackwater/Xe/Academi.
The Hawking story was reported by many news outlets as “Hawking Thinks Black Holes Don’t Exist” or some similar variant. Here is the Smithsonian’s headline, Space.com’s & Business Insider. So, it’s no wonder people think science is nuts. Often it’s so exaggerated in the headlines and reported as if there is conclusive consensus that when people see otherwise, they get jaded. As soon as I see headlines like the above, I just ignore them until scientists say something. I may not have all the scientific answers, but I know when something sort of looks like BS. Sadly, many people lack this essential skill.
Isn’t all this a bit harsh with medical studies? After all, we do have medicine that works. Contraceptives that stop pregnancy, chemotherapy for cancer treatment, treatments for AIDs…
And my latest surgeon pointed out that no wifi was used, the machines had APC, and a mission critical OS. (all the clamps and scalpels were laid out on a table – just it case.)
The drugs they sent me home with where wonderful and basically side-effect free.
Sub
I think that part of what is going on here is statistical illiteracy. If we use a p = .05 rejection region and we do a lot of experiments, then we’ll get a lot of false positives.
Unfortunately many of these false positives are announced with screaming headlines only to be retracted.
You see a LOT of this if you have any encounters with the rabid anti-GMO “activists”.
Worse than that. Reporting bias skews the number of problematic results. While 1/20 of p<.05 tests are expected to be false positives, if you test 100 variables and only report results significant at the 0.05% level, leaving the rest unpublished, it's fairly likely that all of them are bogus.
I expect that phenomenon plus the fact that there's money in positive results go a long way to explain the problem with drug tests.
Yes. An example is where researchers look for natural pharmaceuticals in, say, sea sponges that shrink tumors in mice. By testing about 1000 of these, they are going to find a few positive results simply b/c of the huge sample size and use of a p-value.
You can always use a sequential Bonferroni or Dunn-Šidák adjustment of the acceptance level for this problem.
To say nothing of proper study design in the first place. State your hypothesis, check the results, don’t just go looking for effects. I believe this is known as “doing statistics by the stars”, do ALL the correlations and see where the p values are starred as significant. Very bad form.
I hadn’t heard “doing statistics by the stars” before but that’s a corker, thanks.
I will definitely use that next time I lecture our students on experimental design.
Aren’t there some journals moving towards “preregistration” so that one can keep track of the “file drawered” studies?
“While 1/20 of p<.05 tests are expected to be false positives"
The fraction of p<0.05 tests that are expected to be false positives is not 1/20. That is the fraction of false hypotheses that are expected to produce a false positive. The fraction of P < 0.05 tests that are false positive is a function of the fraction of hypotheses that are true and the power of the test. If all hypotheses tested are really true, there are no false positives. If all the hypotheses tested are really false, there are 100% false positives. Reality is somewhere in between and I think that is what Ioaniddes and Leek are trying to model.
One quibble: Ionnidis (and the others) are not really dissing science, but rather dissing the lack of adherence to proper scientific principles. Unlike Bachmann, he is not proclaiming that revelation from g-d is superior to human knowledge. The call is for *better* science, less bias to studies, greater power of trials, improved protocols.
Exactly. I was going to bring up this point too, because Ionnidis was used by a lot of clueless alternative medicine advocates, who seemed to see his criticisms as either default support for alternative medicine or endorsing the futility of having any objective measure at all, so anything goes. The tu quoque argument, also known as “oh yeah, but you’re just as bad! If you can do it so can I!”
Someone tried it on me. I told them that Ionnidis was, for the most part, counted as a skeptic and advocating even stricter scientific standards. If he thought mainstream medicine was failing to be scientific enough, then examining the studies coming out of the Alt Med community would cause his head to spin like a furious top.
Agree 100%.
IMO, the almighty dollar is at the root of at least some of this.
Drug studies are funded by companies that have a financial investment in the results, thus causing a fair amount of fudging.
And, newspapers, websites, and other news outlets scream in absolutes to get hits so that they can charge their advertisers more. Those hits aren’t going to materialize over lukewarm descriptions; the content has to scream. L
Absolutely true.
Every piece of news has now become an absolute: the most revealing…, the upending of conventional wisdom…, a radical revolution in…, the most extraordinary event…etc. Our media are a kingdom of superlatives. Every piece of news has become extraordinary. We live in times where only extraordinary things happen. Any news from the scientific establishment gets magnified beyond common sense.
The town crier of ol used a powerful bell to be heard over the humdrum noises of the street, while shouting “Oyez” (meaning “Listen, you”), and when he finally got silence, he could read his announcements, but always with a loud voice.
In our modern world, the flow of information has turned into a maze of cac ophony. Attracting attention requires forcing it with overwhelming claims.
However, it’s not even sure that this is a modern phenomenon.
Amazingly (yep), Proverbs (the book of the Hebrew bible) opens up in its first chapter with the “Call of Wisdom in the streets” haranguing the crowds. Wisdom is always a woman in the Hebrew Bible. (she will become Jesus in the Christian New Testament.
To get their attention she announces the terrible fate awaiting the simpletons and fools that they are.
The drama of Wisdom is that they all hear her, but nobody listens to her.
Proverbs 1:
THE CALL OF WISDOM
20 Wisdom cries aloud in the street,
in the markets she raises her voice;
21 at the head of the noisy streets she cries out;
at the entrance of the city gates she speaks:
22 “How long, O simple ones, will you love being simple?
How long will scoffers delight in their scoffing
and fools hate knowledge?
23 If you turn at my reproof,
behold, I will pour out my spirit to you;
I will make my words known to you.
24 Because I have called and you refused to listen,
have stretched out my hand and no one has heeded,
25 because you have ignored all my counsel
and would have none of my reproof,
26 I also will laugh at your calamity;
I will mock when terror strikes you,
27 when terror strikes you like a storm
and your calamity comes like a whirlwind,
when distress and anguish come upon you.
Perhaps I will be allowed to post the end of this remarkable (yep, “extraordinary”) chapter of Proverbs 1.
It’s a paean to the need for knowledge.
Of course, in the perspective of the writer, this knowledge is of religious source, and comes from the “Lord”.
Female Wisdom is his envoy to the credulous Jews, who, far from being “monotheists” were also fond of superstitions and irrational beliefs.
The greatness (in our view) of the ancient Greeks is that, although they had the same belief in the value of knowledge, by contrast to the ancient Hebrews they started seeking it in the understanding and explanation of the physical world.
The two ideologies were poles apart.
28 Then they will call upon me, but I will not answer;
they will seek me diligently but will not find me.
29 Because they hated knowledge
and did not choose the fear of the Lord,
30 would have none of my counsel
and despised all my reproof,
31 therefore they shall eat the fruit of their way,
and have their fill of their own devices.
32 For the simple are killed by their turning away,
and the complacency of fools destroys them;
33 but whoever listens to me will dwell secure
and will be at ease, without dread of disaster.”
To be sure, there’s lots “worng” with science these days. Profit-motivated medical research is a mess; the journal-based peer-review process is having a lot of trouble adapting to the Information Age; and science reporting generally spins everything into a cure for cancer or a solution to the energy crisis.
At the same time, more is right with science than at any time earlier in history. These problems aren’t new and we’re dealing with them better than ever before, even if we really, really need to figure out a way to finally slay these dragons.
So, yes, it’s very important that we pay attention to the problems and work at fixing them.
But a bit of perspective is called for, too. And said perspective should reasonably make these not-to-be-trivialized problems rather small in the big scheme of things, smaller than they’ve ever been, and shrinking.
Cheers,
b&
Or maybe it’s more that that’s lots wrong with human systems that motivate behaviour that isn’t always good for society. Science as a tool is good but the executers of that tool abandon key processes of that tool in pursuit of other goals (ahem Wakefield).
I’d have to disagree with you there. The profit motive (as imperfect an incentive as it is) has done more for applied science than governments and non-profits ever have. The majority of medicines available to us today are a direct result of for-profit research and those companies have put billions towards science and filled scientific libraries with tomes of data. The flat mini-computer/cell phone in your pocket, aircraft engineering…need I continue?
What’s new? You have to kiss a lot of frogs before you find a prince, but science is the only reliable way we have to tell the difference.
I think it’s great that science is developing and is under constant criticism (as in critical growth)- but I am sick of people climbing all over pointing holes in things just because they want to be right about something. spare me. I do find very interesting that definitions are being revised especially since my early days of wanting to a physicist growing up! I think I’m going to look into this more thanks to your post!
Ah, yes, ‘The Truth Wears Off’. I had read it when it came out, and was stunned by how terribly biased it was. It truly does focus on the narrow branches of medical research that have the worst history of conflicts of interest, then tries to paint science in general with the same brush. Another example of popular science journalism where the writer seems to just want to create a buzz and get a reaction. Enlightening the populace is not their goal. I see this is still going on.
P.Z. posted a nice rebuttal to the ‘decline effect’ here, laying out the reasons for why science goes wrong, and how it really does fix itself.
Reblogged this on Mark Solock Blog.
Good discussion.
There’s an apocryphal story about a man who has cancer. His oncologist says “I’d like to try this promising new chemotherapy drug.” The man sighs a bit and somewhat wearily says “Yes, doc, I’d like to try it too, while it still works.”
This morning I made a bet with myself: the time interval between the latest, brilliant spoof by Andy Borowitz going online and it becoming a WEIT topic would be measured with an egg timer.
Well, almost.
The Ioaniddis paper is now a classic, and one I’m always referring people to since I’m no longer teaching medical statistics courses. The quality of statistical reporting and analysis in many medical studies ranges from dubious to dismal. If we wonder why, we need look no further than a recent example: Nature editor Henry Gee’s grasp of the p-value.
But. This is not the whole story. If the quality of medical science and research were as poor as that of medical statistics, we’d be dying in droves. We are not. So the practitioners of medical science must be doing something right, even if they don’t always know how to measure, report and evaluate it.
I’m the living proof of it.
Last week I underwent major computer-assisted cancer surgery. The computer-assisted robotics part was essential.
This kind of operation would have been science fiction twenty years ago, cutting-edge experimental ten years ago.
Now I’ve not only survived a procedure that would likely have killed me if performed by any other means, but also recovered sufficiently to be able to tell about it.
I call that progress in applied science.
Congratulations on your successful treatment.
It’s a bit odd to think that you’re living in a science-fiction world, isn’t it?
Yup. If only it were a bit more evenly distributed.
It’s odd that I’m still living, period.
Over the past half-century, I should have died on at least four different occasions, had I been born a mere decade or so earlier. As I said, I’m a living proof of medical progress.
On the other hand, the “science fiction” aspect of it is not odd at all. The term became popular in the 1920s, if memory serves.
Before that, Jules Verne described his works as “romans d’anticipation”, anticipation novels. Very wise. Astute SF writers are hard-nosed trend hounds. Anticipation is the courage of non-linear extrapolation.
Yes, congratulations and very glad you’re still here.
So many of the people who complain about “Western allopathic medicine” would have been dead before their present age had they been born 100 years ago.
Sounds like Mr Occam is happy to have gone under scalpels designed or improved by the application of his namesake razor.
Glad to hear that the treatment is going to plan.
Another basic confusion is that there’s a difference between, for lack of a better phrase, “basic research,” and some of the studies that are cited as “failure of science” (as dubious an interpretation that might be). “Basic researchers,” say in molecular biology are trying to understand causal “mechanisms,” while these latter studies are about “effects” (without necessarily understanding mechanisms). Now these can lend plausibility to hypothesized mechanisms, but are not substitutes (for understanding them). Finally discovering/finding out mechanisms is much, much, harder and takes much longer. The statistics part is a red herring; that’s confusing methodology with epistemology. There could be many, many reasons–not necessarily having to do with the incompetence of researchers–why “strong” statistical generalizations may not emerge in medical research…
“We also know the speed of light in a vacuum is invariant, and we know that speed to an accuracy of one meter per second.”
Actually the speed of light is now known EXACTLY because it is a defined quantity. c = 299,792,458 meter/second. (A great deal of confidence in Maxwell and Einstein!) Then the meter is defined in terms of the second because, with atomic clocks and all, we can measure time more accurately than distance.
http://en.wikipedia.org/wiki/Speed_of_light
I always redefine c = h = G = 1. It makes physics much easier.
I do wish that journalists would not confuse the existence or nature of the event horizon with the existence of black holes. Black holes exist and are observable.
Hawking is quoted as saying “There is no escape from a black hole in classical theory, but quantum theory enables energy and information to escape.” This is a reversal of his earlier position that information was lost in a black hole. It has nothing to do with them not existing.
The book by Leonard Susskind, “The Black Hole War”, is a good resource for some of the background of these developments around the theories of black holes.
Well, yes, each of those constants can be 1, but you can’t write c = h = G, because dimensional analysis.
/@
PS. h = 1 or h/2π = 1?
George Johnson wrote a book about Nobel Prize Winning physicist Murray GellMann which the latter found to be less then satisfactory. It is my understanding that the consensus is that there is much misinformation in the book.
It is also my information, via Johnson himself, that he and GellMann are no longer on speaking terms.
I enjoyed that book.
I wish I could remember where, but I thought I read that studies are often not replicated not because they’re wrong but because there’s much more caché in publishing the original study and doing never-before-attempted research. Can one get tenure validating previous researchers’ results?
First, a few corrections: Ioannidis’ paper was published in 2005, predating the Lehrer article in the New Yorker. Ioannidis is not a professional statistician, neither a practitioner nor a researcher. He is a professor (on leave) in the Department of Hygiene and Epidemiology at the University of Ioannina School of Medicine, as well as an adjunct professor at Harvard in the Dept. of Epidemiology, and a professor at Stanford in Disease Prevention, in Health Research and Policy and, *by courtesy*, in Statistics. I don’t want to diminish those I’m sure well-deserved titles, but it bears repeating that Ioannidis is *not* a statistician.
However, his famous (infamous?) paper “Why most published research findings are false,” keeps popping up. It’s sensationalist and misleading title is undoubtedly like catnip to those scouring for a scientific “controversy.” I have read Ioannidis’ paper as well as several critiques of it. It does not show that most published research findings are false. In the words of Jeff Leek (Biostatistician at John Hopkins), Ioannidis’ paper shows that “under assumptions about the way people perform [hypothesis testing] and report them it is possible to construct a universe where most published research findings are false positive results. The paper contains no real data, it is purely based on conjecture and simulation.” This is a far cry from the claim that “most published (medical) research is false,” yet it keeps getting reported that way by people who would rather use a poorly titled paper to help them diss science rather than actually read the paper themselves.
Leah Jager (Math department, US Naval Academy) and Leek have an interesting paper from 2013 that actually purports to show that only about 14% of reported discoveries in the medical literature are false. Their paper is interesting because it looks at real data, looking at reported p-values from 77,000+ medical papers from 2000 to 2010. Ultimately though, it too can be interpreted/criticized as “merely” a paper that shows it’s possible to construct a universe where this type of thing happens. Their universe is much closer to ours than Ioannidis’ however; it’s good work. See the paper here: http://goo.gl/T7gp27
Personally, I think I tend to come down closer to Andrew Gelman’s take on the matter (he’s a famous statistician at Columbia). He claims that the whole exercise of trying to assess what the distribution of correct results is like from things like p-values is ultimately hopeless. See his discussion here: http://goo.gl/p6PQA4
It’s a very interesting issue all around, one I come back to and toy with again and again. I just wish I would stop seeing various science journalists misreport what is actually known about the issue.
Correction: the Jager and Leek paper looks at 5322 reported p-values from medical paper abstracts.
As I commented over on the post you kindly recommended, a new journalistic report[1] about Dr. Ioannidis makes clear that he is very much a fan of science. Ioannidis is simply a working scientist who tries to remind scientists not to make claims beyond what their data show. Another working scientist who has interesting publications about “researcher degrees of freedom” and “p-hacking” is Uri Simonsohn of Penn[2], who with his colleagues writes thought-provoking (and sometimes laugh-out-loud funny) peer-reviewed articles about statistical and other methodological errors in published research. The papers from that research group are very well worth reading.
[1] http://www2.macleans.ca/2014/01/17/when-science-isnt-science-based-in-class-with-dr-john-ioannidis/%C2%A0
[2] http://opim.wharton.upenn.edu/~uws/
That Maclean’s article gets it just as wrong though, and they are simply parroting what Ioannidis says (to a room full of doctors and one journalist) rather than reading his paper themselves, or conferring with people who have read his work.
They report that in his paper, he “described how a combination of uncertainty and publication bias creates a maelstrom of spurious findings that don’t hold up to scrutiny over the long-term.” This is not true. Ioannidis’ paper showed that in a very particular type of hypothetical universe it is possible that most research findings are false. And his paper does zero sensitivity analysis on the rather large and unrealistic assumptions of his very simple model. Nothing in his paper is based off of any data. (See my comment above.) The Maclean’s article fails to report any of this.
Ioannidis is most certainly a friend of science, but he bears the blame for his irresponsibly titled article, and especially for continuing to puff the article out as saying much more than it actually does. This is damaging. Arguably, his paper’s greatest contribution in the interest of science has been a negative one, giving those who want it credible-looking fodder that says “science is wrong” in some way. For a great example of this effect, have a look at the comments on the Maclean’s article where several people use Ioannidis’ claims, parroted through the agency of the journalist writing the article, to buttress their climate science denialism.
Agree with most of the comments above. Ioannidis’ title is over-dramatic but he’s making an important point and a lot of work is now being done on meta-analysis and modelling and correcting for publication bias.
As Jerry says, most of the valid criticisms are in medical areas, where publication bias seems most serious – see Ben Goldacre on this and work to overcome it:
http://www.badscience.net/
And just a reminder that Richard Feynman made many of the same points, especially referring to psychology and related disciplines, 40 years ago in his classic Cargo Cult Science lecture/essay:
http://neurotheory.columbia.edu/~ken/cargo_cult.html
Thanks, interesting and educational discussion.
Since I didn’t hold Lehrer high after reading a few pieces, I never got taken in by his later columns. And I think that piece was one of those he got roundly criticized for.
“Since I didn’t hold Lehrer high after reading a few pieces”. It strikes me that I can’t prove that, so strike that.
A bit late to this thread but check out NIH Director Francis Collins acknowledging the growing errors:
http://www.forbes.com/sites/matthewherper/2014/01/27/nih-promises-to-make-scientific-studies-less-wrong/
Thank you for that link! I hadn’t seen that article. But note the infuriating irony: the Forbes article trades helping the NIH communicate its message for “making provocative statements [at the expense of] presenting technical details,” one of the very things the NIH is so decrying.
The title of the article refers to “scientific studies.” The first sentence: “…many scientific studies are, well, wrong….” And “they [the NIH] insist that science is still ‘self-correcting’.” But click on their link and read what the NIH actually says, about “restor[ing] the self-correcting nature of preclinical research.” And the first sentence: “A growing chorus of concern, from scientists and laypeople, contends that the complex system for ensuring the reproducibility of biomedical research is failing….”
Biomedical, preclinical research is a very specific and relatively small subset of “science.” Why is this so hard for journalists to understand, appreciate, and/or commit to communicating? Incredibly irritating!
Anyway, thanks for the link!
My spouse and I stumbled over here coming from a different
page and thought I might check things out. I like
what I see so now i am following you. Look forward to
looking at your web page again.