There’s a new Pew survey out that asks a timely question, or rather several timely questions. How often can Americans distinguish between factual statements (that is, statements that can be empirically verified or disproven) and statements of opinion? And does that depend on whether the statements are congenial to their ideology? Does exposure to or trust in the news media help you distinguish between fact and opinion?
You can find a summary of the survey (5,035 adult Americans, 18 or older) by clicking on the screenshot below, and the full pdf is here.
Here are the five statements of fact, five statements of opinion, and there were two “borderline” statement that were mixed: part opinion and part fact. (This last group wasn’t subject to as much analysis as the first two groups.)
And here’s what the respondents were asked; remember, a “factual” statement simply makes a factual assertion—it doesn’t have to be true:
The results were that most Americans (71%) were able to pick out at least three of the five factual statements, but only half (50%) were able to correctly distinguish four or more of the factual statements as assertions of fact. People did a little better with the opinion statements: 78% were able to correctly classify three or more as opinion, but only 59% of Americans got four or more of the opinion statements correct as being opinions.
Overall, only 26% of all the respondents correctly classified all five factual statements, and 35% correctly classified all five opinion statements. This is a bit disheartening to me, as the distinction above seems pretty clear (I’m ignoring the “half factual/half opinion” statements). However, academics or scientists might be better trained to distinguish fact from opinion, as the former are the ones that are empirically testable.
As the chart below shows, people judged to have high political awareness, digital savviness, trust in the news media, and interest in the news, were (with exception of opinion for the news hounds) better able to classify a statement as fact or opinion.
Further, both Democrats and Republicans were more liable to classify BOTH factual and opinion statements as “factual” when those statements were congenial to their political ideology. This graph shows that as well. Look, for example (bottom half of figure), at how much more often Republicans classified the opinion statement “illegal immigrants are a very big problem for the U.S” as factual than did Democrats. Conversely, more Democrats than Republicans saw the “we need to increase the federal minimum wage for the health of our economy” opinion as a statement of fact.
There’s a lot more to this survey than the brief summary I’ve given here, but these are the main results. The upshot to me is that Americans are worse than I thought at distinguishing between fact and opinion, but they’re not hopeless. And when a statement is an opinion, both Democrats and Republicans are more likely to see it as factual if it’s ideologically appealing (there’s no real difference between the parties in their overall error rate here).
It would be interesting to ask other questions as well, like “Hate speech (speech that denigrates religion, ethnicity, or national origin) is prohibited by the U.S. Constitution” (a factual statement, though a false one), and “The Constitution allows all Americans to own handguns to protect themselves.” (Another false factual statement; not all Americans are allowed to own guns.) You can also invent your own questions.
Upping the percentage of people who can distinguish between facts and opinions could in fact be a very useful goal of a “critical thinking” course. In fact, it might be the very first exercise in such a course.
sub
I’m underwhelmed. The statement about “essential to the health of the US economy” is either facrtually true or factually false if we have a working definition of “essential” and “health”; otherwise it is meaningless. And it is factual (and false) that voter fraud has had any effect worth mentioning on US elections (unless you include as fraud the shenannigans in Florida 2000, in which case the statement is factual and true).
So I am proud to say that I would have flunked this test.
I would also have failed. Statement about the effect of the minimum wage or immigration are conceivably provable or (more easily) disprovable.
I had the same problem with “Government is almost always wasteful and inefficient” and “Immigrants who are in the U.S. illegally are a very big problem for the country today.” Both of those statements seem to me testable in ways that “Democracy is the greatest form of government” and “Abortion should be legal in most cases” are not. I would have flunked too, I’m afraid.
In fact, assuming that the researchers are mostly likely liberal and Democrat, their putting those last two items in the opinion category bears out the result of the study—that one’s ideology leads to mistakes about what’s fact and what’s opinion.
“…if we have a working definition of ‘essential’ and ‘health’”
The point you’ve made – fact/opinion distinctions are sensitive to assumed meanings – is true, but that’s a mighty big ‘if’, and who is this ‘we’? I might have my definitions, but how many would agree with my definitions?
+1. But I’d go further and suggest that both the location of the fact/opinion divide, and its very existence, is a matter of opinion.
We did truth tables and symbolic logic in a critical thinking course I had once – it was very neat. A science course had us, for our very first class activity, work on distinguishing between normative and empirical statements. Honestly, after taking that symbolic logic course I was all fired up thinking that it wasn’t a part of standard middle school curriculum. I don’t see why it isn’t.
A lot of the problem is that word fact. Facts are true. Pew assumes that factual does not mean true, but that isn’t as clear as it needs to be. Synonyms listed include truthful, true to life, accurate. So sometimes factual means what Pew is trying to test and sometimes it implies truthful. This is a serious confounding factor for this survey I think.
It would be better if this were explained more clearly, and if there were questions to test this definition is what people are using.
I agree. Surveying this kind of thing is notoriously difficult.
In many contexts (see any logic text), “fact statement” carries no indication of truth value.
This is (sometimes) a stipulative definition. People do have trouble with those.
Early in the 20th century there were a lot of philosophers (and some as late as Chisholm, if I recall) using “fact” to mean a proposition, which is *really* confusing.
I spent 22 years as a Unitarian Universalist, and a major concern of mine was a widespread inability to distinguish fact and opinion.
This became evident when on one Sunday morning we had a guest speaker who gave one of the worst talks/”sermons” I had ever heard, bad because it got all kinds of facts wrong. When I raised objections during the social hour, just a few too many people told me that these objections were simply “your opinion”. They were wrong.
My first thought reading this article was exactly what was on display in the story you relate here. The pervasive “everyone’s entitled to their own facts” point of view that seems to have infected nearly all demographics in our society. From hippies to Fox News addicted Boomers. From woke college students to IC Neo Nazi punks.
These days if you try to demonstrate that the claims someone makes in support of their views are factually wrong you are treated as if you are engaged in a morally reprehensible behavior and infringing their rights.
Also, in post truth world, you can easily doubt everything. In setting up people to distrust all sources of information, you make it easier to get them to accept your BS. Trump basically stole a page from Putin’s playbook, who stole it from various dictators past & present.
Bunge wrote a paper for Skeptical Inquirer years ago: “Absolute Skepticism Equals Dogmatism”.
Basically the idea is that everyone has some filters, so being absolutely skeptical on paper makes one skeptical of random things, including ones that should not be so doubted (like the external world!) …
I recently read comments below a report about unvaccinated children getting infections. An antivaxer commenter accused pro-vaxers that they are gullible people indoctrinated by the vaccine industry as well as by “the cult of the globe”.
My parents are UUs, and growing up that’s one of the problems I had with them too (the congregation – not my parents, who are more “hard nosed” than that).
We’re fucked.
Fact or opinion?
Fuckual opinion!
Neither, since “fucked” is too vague to determine anything.
I wonder how much misunderstanding of the statement has to do with the outcome. I suspect there is much of this associated with the questions. Also, people who do stay up to date with the news are becoming less and less which makes many of the outcomes little more than guesses. Example: Half the population of adults could not point out on a map where Iraq or Syria were. So the question about ISIS would get an answer that would simple be a wild guess. Maybe before accepting a person’s answer they should be asked if they are properly understanding the question and are they familiar with the subject.
Maybe I just don’t like surveys. The question about Obama could simply be a measurement of stupidity. I give you our current president.
I agree with you in a way, but the purpose of the survey was to see if respondents could distinguish between factual statements [both true & false ones – it doesn’t matter] & opinion statements – the respondents don’t need to be in possession of facts to do this – in theory. The respondents don’t need to know where Syria is nor who or what ISIS is.
I think the test should have used fictitious news: “The dumblwots beat the illanciars at the battle of gramble-weed-snot in 1321 AD”
And fictitious opinions:
“knoberry sauce is the only sauce you should squeeze onto a danglewurst dog”
I think that would have been easier for people to distinguish fact statements from opinion statements. But then they couldn’t have judged other things like political bias, which is just as interesting (to me anyway).
Each respondent was asked supplementary questions to establish their position along the spectrum for politics, education etc. But this test doesn’t seem to have a way to distinguish clearly between language comprehension & political bias.
Perhaps they could have produced two questionnaires & tested the respondents randomly on one or the other.
That said… I tried to construct an opinion statement about contemporary politics, economics, morals [say abortion] & I wasn’t satisfied that my attempts were clearly classifiable by others [or even myself!] as an opinion statement. Give it a go HH.
It’s the sort of thing I like doing, but if Pew’s experts couldn’t (and imo they didn’t), I’m not sure I’d do any better. (I wish I’d seen your comment yesterday because I was looking for something different to do.)
IIRC, Susan Jacoby reported in her “The Age of American Unreason” that approx. 25% of college educated 25-34 year-olds couldn’t locate Syria on a map with the names of the countries on the map. (Is it a matter of not knowing or of being lazy?)
A question that I would have liked to have seen included: “President Trump’s (now ex-) White House security chief is bullet-headed. Fact or opinion?”
I mention that because a NY Times article stated it. Perhaps it is a fact. If so, I demand that the shape of the head of every person mentioned in an NYT news article be similarly described. (And I want to know the shape of the heads of everyone associated with the NYT.)
I remember several years ago the Times reporting on a Chinese space accomplishment (landing a probe on the moon?). Is anyone shocked that the Chinese government, and its citizens, would make a big deal of it? In this regard the Times referred to Chinese “propaganda.” I don’t recall the U.S. media ever referring to U.S. “propaganda” when the U.S. has tooted its “Exceptional” horn. Apparently it’s not propaganda when we do it.
Critical thinking teachers have noticed that completely fake examples in various contents actually reduce performance because they distract.
Students have to learn to be content-free in reasoning (formal only) and that’s *hard*.
“contents” -> “contexts”
Well, the question about ISIS losing its territory in 2017 is a question of fact, whether one knows the answer or not. It wasn’t a question of whether the statement was *true*, though conveying the difference to the subjects is often difficult.
Like ‘There were 4 million sheep in New Zealand in 1951’. One doesn’t have to know the answer, or even what a sheep looks like, to identify that as a question of fact, whatever the actual number was, or even if anybody counted them.
‘There were too many sheep in New Zealand’ is equally obviously opinion.
As it happens, even the ‘factual’ statement about ISIS contains an implied judgement – ‘ISIS lost a significant portion of its territory…’ – what is ‘significant’? 5%? 20%? 70%? I’m assuming that the actual percentage is indeed what most people would regard as ‘significant’, thus making the statement factually decidable. (Though the losers might hairsplit, like the Black Knight in Monty Python. ‘Tis but a scratch’)
cr
(I made up the 4 million, by the way)
It is still a factual statement (about “significant”). It may be one that’s too vague for a truth value, or whatever one’s view on *vagueness* is.
Those who are less interested in the news are better able to detect an opinion.
I was right to quit reading The Guardian.
In my previous life as a harried HS English teacher, I taught a critical thinking unit to Jrs & Srs. Indeed, one of the first exercises in that unit was fact v opinion–and especially the distinction between true and false facts. You can see how that turned out. Sigh.
Most definitions of the word “fact” include this:
“something that actually exists; reality; truth”
Thus, there is no such thing as a false fact. By definition, all facts are true. It is redundant to say “true fact” and it unfortunately implies that there is such a thing as a “false fact.”
All righty then. If say, Bill Maher said, “Trump is an orangutan,” that would be an opinion? Please note–I am not saying that Bill Maher would ever say that. Just sayin’.
Actually that would be joke. And even better Trump sued him for saying it.
That would be a factual statement, a false one.
OR
A joke, which is not a statement, since it cannot have a truth value.
Maybe so, but there IS a difference between the noun (fact) and the adjective (factual). It is the latter the poll was testing, not the former. So while you can make the case that there are no false facts, a factual statement could be either true or false.
I agree about the adjective, but it is highly confusing. For example, how many people would catch the difference between saying a certain statement is a factual statement and saying a certain statement is one of fact (true)? I think that those who study logic in a college course would know that a factual statement (as opposed to an opinion) may not necessarily be true, i.e, a fact, but not so many in the general public.
I agree too. It’s a very clear distinction, but one that is often hard to make. I often have trouble explaining the distinction.
‘What is the age of the earth’ is a question of fact – even if there are many different ‘opinions’ on what that age is. One figure must be the correct one, even if we don’t know it for certain. (Logically, everybody could be wrong).
What happened to Amelia Earhart? is a factual question. We just don’t know the answer for sure. ‘She crashed on Nikumaroro Island’ is a statement of ‘fact’ which may or may not be correct. (We need a different word for ‘statements which purport to be factual (but may or may not be correct’)
Incidentally, I would take the phrase “a factual statement” as meaning it is true, which is the opposite from what you (Historian) use it to mean. One of us must be wrong 😉
In fact (umm, sorry) the meaning of that phrase doubtless depends on local conventions. It may mean something different to logicians from what it does for the public at large.
cr
It sounds to me like we need to coin a new word to replace factual, because everyone is going to think that factual means fact, and that fact means “something which has been proven to be true.”
The one about raising the economy threw me because I believe it has been tested, and there are economic models that prove that we need to raise the minimum wage to prevent various undefined problems. I guess this comes down to whether you think economics can ever be like science, and whether you read too much into the question as I did. (If you were to start the survey with clearly fictional examples you would prep people to think more objectively and they would answer the rest of the survey differently.)
No, all facts are what is the case. Facts are neither true nor false; *propositions* or *statements* *about* facts are.
(The poll uses this more or less correctly in the sense of standard usage amongst logicians, etc.)
Reblogged this on The Logical Place.
I don’t trust this USA Pew survey – some portion of the respondents would find the wording too hard to process – for them it’s a test for reading age as much as anything else! According to Readability the average reading age in my country [UK] is nine years old [yes they are biased towards demonstrating a low age, but it’s a similar result to others I’ve seen]
In fact, in the .pdf provided by The Ceiling Cat, I find this on page 45:
so there’s an awareness by Pew of the problem of comprehension & it goes on to explain what they did to make it more understandable, but to me the reading level is The Guardian [14 yo] & not The Sun [8 yo]
I’d agree to a certain extent. I think the reading age for this survey is higher than usual, and they could have done a better job of making the questions easier to understand. Some are quite complex, and would take a bit of thought for many people.
The point that Jerry talks introduces at the end (factual statements that are wrong) is, I think beyond most people without specific education on the topic. It’s like the difference between a theory in science and the general understanding of a theory. A fact is true, and the confusion between that and a statement that is factual but false would be too big a leap for this sort of survey. I suspect I would label a false factual statement as opinion in a survey like this.
Fromage frais et bonnet de douche, mois Martin Amis Heather!
[borrowing freely from the Peckham entrepreneur & polymath Derek Edward Trotter]
I wonder if there was an option for a survey taker to say words to the effect, “I don’t have enough information/evidence to enable me to determine whether this is fact or opinion”?
this is the .pdf from Pew with all that info [linked in the OP too]
It is clear from the appendices that responders were given a binary choice only. A very small number of responders left some questions blank – in error or on purpose.
This doesn’t surprise me in the least, especially given the religiosity of the US in general.
For example, the core of many of the arguments most of the commenters on my website and I had with a particular, now-banned, commenter was that he was unable to understand that his main contentions (the Christian God is real; God = truth) were opinions and not facts.
His opinions about the existence of God informed his whole worldview. It was clear that most of those he normally communicated with thought the same way he did. It made it impossible to debate with him because he couldn’t even get his head around the fact that his belief that God was real and was Roman Catholic, wasn’t fact but opinion.
If God is real, and everything God says is the truth, then if your religion teaches that abortion is wrong, that becomes fact. Everyone who says otherwise is wrong. Opinion doesn’t come into it.
And, if you get these lovelies that get all their information from the fox it is all made up opinion. They then recite same to you in the comments as if they came down from heaven. How many times I get this – they have to cross at the appropriate crossing or they can’t declare asylum. So many are not crossing at the correct place. Holly shit, if you would watch another channel you would find the guards are purposely not allowing them to cross at the correct location so they cannot declare asylum.
In international law, they can still ask for asylum wherever they cross too. As you say, it’s nasty, petty, small-minded people taking advantage of having a little bit of power to hurt and humiliate others.
It doesn’t surprise me either, but I also would not be surprised to find that people today are better at distinguishing factual statements from opinion than they would have been in the past.
This finding is soft* and represents a single point in time. IOW, interesting but of little use.
*only 12 questions were asked and some here have already pointed to problems with them.
God aside, this is an interesting question not addressed by the poll: moral realism.
If you think that
“Abortion is wrong”
has a truth value (even if “false”, like some contemporary ethicists), then “Abortion is wrong” is a factual statement.
When I teach logic and critical thinking to CFI groups, I mention the debate over moral realism and then say that moral language is too much for a first such course and we’ll move on. 🙂
It’s really quite simple, but then I am not American!
The facts are either right or wrong demonstrable and provable with statistics, mathematics, census form data etc. The opinions are not.
You would not have got 100% on the test then for the reason given in comment #2 Paul Braterman
You are also wrong philosophically – a statement of facts is not necessarily “provable with statistics, mathematics, census form data etc.” because we do not have access to all facts.
I wonder how many people answer poll questions with a level of sophistication not accounted for by the pollsters. Everyone knows that fact beats opinion in an argument. They also know how the results of these polls get reported in the media. Is it much of a leap to imagine that the pollees feel motivated to report an opinion as a fact, or vice-versa, based on a perceived advantage to their team, red or blue?
That may be frequently true.
A sort of “I feel moderately about X but I’m going to rate it 10 just to make sure the poll comes out in favour” sort of thing.
cr
Here is an excellent test that everyone should take. It determines if you have a good assessment of what things you think you know for certain versus those things you know you’re not totally sure about.
https://rq-test.meteorapp.com/rq-test/detail/d222cfcc-v3b0-43fd-lj9b-1167011dc01b
it requests an email address for some reason & no info about who they are [no about page etc]
It’s been around for a long time. I think you can just make up an address. Seems like it might be broken though.
I made one up, but it was broken anyway as you say
Sadly it looks like the basic test, which is free, is down. Here’s the comapany “about” page: http://www.projectionpoint.com/index.php/menu/about_us
Since it seems to be broken I’ll write some more about it. It’s an ingenious test that presents apparent statements of fact. For each one you rate 1 to 10 how confident you are that the fact is true. 1 means you are totally sure it’s false and 10 means you are totally sure it’s true. 5 means you really have not idea if it’s true or not.
Then is rates how accurate you were about your confidence in your own knowledge.
Obviously if you rate everything 5 they can’t produce a score.
Thank you. It is also about our ability to assess risk I think, right? Pity it’s not working.
Facts, schmax — truth, schmooth. Next thing you know you’ll all be complaining that no one can tell the difference between a valid and an invalid argument. Lighten up. Nothing at stake here.
A few days ago, some people here took exception to my invitation to be skeptical of all news sources, whatever their political leanings, whether Fox or CNN. Here is an interesting website that compares stories on the same subject from various news outlets, the NYT and National Review, for example, for their slant:
The authors of the site might, for example, compare the headline with the content of the article. They might point out differences in emphasis that distort the facts. I think that the analysis is a useful corrective to one’s own bias.
11% of *Democrats* don’t know that Obama was born in Hawaii? (Or possibly alternatively don’t know that Hawaii is part of the USA?)
Under which rock have these people been hiding?
(Also, 37% of Rethuglicans, but what else would you expect?)
cr
Where are you getting Hawaii? I’ve checked twice & I can’t see the place mentioned.
The Pew questionnaire gives us the % who classified the statement: “President Barack Obama was born in the United States” as factual [true or false] rather than opinion:
Dems: 89% [& ‘deniers’ = 100 – 89 = 11%]
Reps: 63% [& ‘deniers’ = 100 – 63 = 27%]
Your seeing of a “Hawaii” that isn’t there may constitute a bias of your own [or I’m going mad] 🙂
I don’t rate this question at all! Obama spent four years in Indonesia growing up & has a significant wing of half-siblings born & raised in Kenya. If I was living in the USA & deeply uninterested in politics I might half-remember that Obama has ‘foreign’ connections & tick the “No” box [after all 6% of US citizens are born US citizens abroad] – thus putting me in the “political bias has marred my reasoning about facts/opinions” category, when in fact I ticked the wrong box independently of being Dem or Repub leaning.
Anybody would have to be living under a rock to have missed the ‘birther’ controversy over the last ten years and not know that Obama was born in Hawaii. Which is part of the USA.
However, I may have succumbed to the ‘true’-vs-‘question of fact’ confusion there.
I took the 89% to be Dems who thought it was true vs 11% who thought it was untrue. Re-reading it carefully, it does seem to be 89% who thought it was a question of fact vs 11% who thought it was a matter of opinion (though how they could ever come to that latter conclusion, notwithstanding the points you made, I cannot comprehend). On that basis, I’m staggered that 37% of Repubs got it wrong too, since the whole of the ‘birther’ argument hinged on that fact.
cr
Your “living under a rock” remark is unintentionally correct – that is how vast numbers of people live despite existing in an era with unparalleled access to information. In the first quarter of 2014 2,555 American adults were asked to name as many US presidents as they could in an online survey by vouchercloud.net – a UK voucher site offering healthy prizes for good results.
10% Americans couldn’t name more than four.
24% Americans didn’t name Obama who had been in the Oval office since 2009 & was the president at the time.
35% Americans didn’t name George W Bush – the president immediately before Obama.
Acquiring knowledge, fact checking it & joining it together into a model of the world is not a highly valued activity. People get by on fuzzy bits of models that they swap with their peers that they haven’t examined at all! They trust their tribe for a supply of ready meal opinion options to bring to an issue – they couldn’t bake an opinion from fresh raw ingredients to save their lives.
Lincoln was most mentioned at 82%
I’ve always had spending money [earning since early teens] & took up coupons as a kiddo when one cut them out of newspapers. I’d shop for Mum & neighbours – keep the extra item in the 2-for-1s for re-sale & pocket the direct money savings [straight into my money box]. That’s unnatural of course since the coupon thing is associated with middle aged [& older] females.
I speculate that the survey is skewed to older women absolutely uninterested in politics, but all the same it’s a little surprising.
That’s the public square – a weird place.
Congrats to Jacinda if you see her out & about – had her own girl duckling today
I disputed one question: it made the statement “group X lost a significant amount of territory” as a factual statement, and to me, “significant” is really an opinion.
Saying “they lost control of Province X” would have been factual, IMHO.
Jerry is right that it is a good topic for a critical thinking course, but I find it useful as the *second* or *third* topic, which are often all coverable in one 1.5 session.
The first topic is usually to distinguish other uses of language: imperative, interrogative, declarative, etc. *Then* we do the distinction between factual and a few others, including *mentioning* the cases of moral and aesthetic realism.
Then vagueness as a concurrent topic, since there is debate over whether factual statements can be vague.