by Greg Mayer

I posted on Wednesday about the new Pew poll on evolution acceptance, focusing on the divergence between the Pew results and those from 2012’s Gallup poll of the same issue. In both polls it is possible to divide respondents into three classes that can be thought of as those that accept “naturalistic evolution”, those that accept “theistic evolution”, and those that accept “creationism”. The Pew poll shows a greater preponderance of the first two (32% and 24%, respectively, with 32% creationist) than does the Gallup poll (15% and 32%, respectively, with 46% creationist). I considered how differences in wording between the two polls might have affected their results, but could not come to any convincing explanation for the disparity.

The Pew poll was also noticed in the general media (e.g. MSNBC, Reuters, NPR, Christian Science Monitor), and most of these have emphasized in their stories the fact that creationism is much more popular among Republicans than Democrats and independents (a plurality, 48%, of Republicans are creationist, while 43% accept evolution), and that the popularity of creationism among Republicans has increased notably since the last time Pew polled this question in 2009 (at that time, a majority of Republicans, 54%, accepted some form of evolution, while 39% were creationist). The headline from the Christian Science Monitor, “Percentage of Republicans who believe in evolution is shrinking”, is representative.

A number of commentators– for example Andrew Sullivan, David Graham at the Atlantic, Zack Beauchamp at Think Progress, Allapundit at Hot Air, and Francis X. Clines and Paul Krugman at the New York Times– have also taken note of the Pew poll, and offered various suggestions as to what has happened to increase Republican support for creationism since 2009. (Only Allahpundit took up the question I found most interesting– why did the Pew results diverge from those of Gallup; he also noted differences between Pew and recent Yougov and Harris polls as well.) The two logical possibilities are that Republicans have shifted their views, becoming more creationist; or that creationists have shifted their party allegiance, more of them becoming Republicans, while those who accept evolution have become independents or Democrats.

Both of these type of shifts, of Republicans to creationism and of party allegiance, could be happening, and both have been suggested by one or another of the commentators. Beauchamp, for example, suggests that out-of-power Republicans are rallying to the “team” position, while also noting that white evangelical Protestants have shifted their party allegiance toward the Republicans. Graham pointed to some evidence that scientists have also shifted, and are less likely to be Republicans now than in the past.

While most commentators have offered interpretations or explanations of the Pew results, Dan Kahan of the Cultural Cognition Project at Yale Law School has questioned the results themselves, and complained about the media’s headlining of the Republican embrace of creationism. His complaints received increased attention because they were tweeted by noted science journalist Carl Zimmer, who wrote that Kahan had “pick[ed the Pew poll] apart”. Jerry, with a nod to Zimmer, has taken note of Kahan’s piece in an addendum to my post.

So does Kahan land any blows on the Pew poll? Well, yes and no. He criticizes Pew for not releasing the full crosstabs on their poll, and on this point I share his frustration. In the Pew press release, there’s a link for the “Full Report“, but this leads to a pdf consisting of the press release plus a subset of the exact questions with answers. It would have been nice to know how party affiliation correlated with naturalistic vs. theistic evolution, for example.

But Kahan then goes on to make two further complaints, neither of which stand up. First, he says that if we knew what percent of Democratic respondents could be classed as accepting naturalistic vs. theistic evolution, this would change the entire cast of the results, from a “ha ha ha!” at Republicans’ expense, to…. ?; it’s not entirely clear what, but it apparently would be “complicated and interesting”. But knowing this about Democrats would not change the main news about party allegiance: 48% of Republicans are creationists. No matter how the Democrats divide on naturalistic vs. theistic evolution, it’s not going to stop Republicans from looking really bad.

The only way to really alter the import of the Pew results on evolution acceptance and party allegiance is if the numbers are actually wrong, not just misinterpreted, and that’s the complaint that Kahan takes up next. Now, both Allahpundit and I considered the possibility that Pew’s numbers are wrong in the sense that, given other polling data, Pew’s numbers may not be good estimates of true public opinion. But Kahan apparently believes Pew’s numbers don’t add up on their own terms, that there is some “logical inconsistency” in the numbers. He arrives at this conclusion by arguing that since the percentage of Democrats and independents accepting or rejecting evolution has not changed, but the percentage of Republicans who are creationists has gone up, then it should be the case that there should have been an overall decline in acceptance of evolution:

And logically, in that case, the % of the U.S. public overall who now say they are “creationists” would have had to gone up–especially insfoar as the proportion of the population identifying as Republican has increased a lot since 2009 …[note: assuming we include “lean Republican” “independents” in the totals, as we should if we are trying to give an accurate senes of partisan identification]. [Kahan’s brackets– GCM]

But according to Pew there has been no change in overall acceptance, leading Kahan to conclude:

So, something does not compute.

At a minimum, Pew has some ‘splainin to do, if in fact it is trying to edify people rather than feed the apptetite of those who make a living exciting fractious group rivalries among culturally diverse citizens.

But Kahan is wrong about three things here. First, the proportion of the population that identifies as Republican has not increased a lot since 2009. In Pew’s own surveys, the proportion of Republicans since 2008 has varied from 24% to 25% (these exact numbers are shown in Kahan’s post).  According to Gallup, Republican numbers are tending down, not up, over the last few years. There is thus no reason to think that the proportion of Republicans has gone up. In Pew’s 2009 survey of evolution acceptance, the unweighted proportion of Republicans was 25%, right in line with its other estimates, so there’s no sign that Pew’s evolution acceptance polls underestimate this proportion.

Second, Kahan says independents who “lean Republican” should be counted as Republicans, but that’s not what Pew did, and Kahan’s wishes as to how he would want them counted does not affect Pew’s numbers.  Kahan is right that a fuller release of data would be interesting, but no inference of logical inconsistency in the Pew data can be based on this.

And third, Kahan has equated the statistical concept of “the same” with mathematical equality. In algebra, if x + y = z, and you increase y while holding x constant, then the sum, z, must increase. This is in outline form the argument made by Kahan (where x and y represent political subdivisions, and z the overall number). But if x, y, and z are statistically estimated quantities with errors of measurement, then it is possible for x to show no significant change and for y to increase significantly, yet z still show no significant change. This would be especially so if the magnitude of y is small relative to z (and the proportion of Republicans is the smallest of the three main political respondent classes). Thus, there is no contradiction between Democrats and independents being statistically unchanged, Republicans showing a significant change, yet the overall result is also statistically unchanged.

The inference that Kahan tries to make is further undermined by the fact that the overall acceptance of evolution is not the weighted average of just Democrats, independents, and Republicans (where the weights are their estimated proportions in the population), but the weighted average of Democrats, Republicans, independents, other parties, no preference, and refused. There are too many unknowns to be solved for; it’s not x + y = z, but t + u + v + w + x + y = z, and thus  knowing just how Democrats (or Democrats+independents) and Republicans have responded is not sufficient to infer the overall response. (I tried figuring out some of these numbers for the 2009 survey from the summarized results by making assumptions to reduce the number of variables, but when I checked my resulting approximations against the actual numbers in the full Pew data set, I was noticeably off.)

So, Kahan is right that it would be useful to have the full data set, but wrong i) that having that full data set would change the apparent embrace of creationism by Republicans; or ii) that it is possible to infer logical inconsistency in the released data. There could be errors in Pew’s calculations of proportions and tests of statistical significance, but that cannot be inferred from the given data.

But is there something nefarious in Pew’s failure to release some of the data, rather than all of it? Kahan clearly thinks there is. He writes:

…this sort of deliberate selectivity (make no mistake, it was deliberate: Pew made the decision to include the partisan breakdown for only half of the bifurcated evolution-belief item) subsidizes the predictable “ha ha ha!” response on the part of the culturally partisan commentators who will see the survey as a chance to stigmatize Republicans as being distinctively “anti-science.” …

Pew lulled those who are making the [“ha ha ha ha ha!”] response into being this unreflective by deliberately (again, they had to decide to report only a portion of the evolution-survey item by political affiliation) failing to report what % of Democrats who indicated that they believe in “naturalistic” evolution. [Democratic results were not selectively withheld; no political breakdown was provided for this question.]  …

Right away when I heard about the Pew poll, I turned to the results to see what the explanation was for the interesting — truly! — “shift” in Republican view: Were Republicans changing their positions on creationism or creationists changing their party allegiance?

And right away I ran into this logical inconsistency.

Surely, someone will clear this up, I thought.

But no.

Just the same predictable, boring “ha ha ha ha!” reaction.

Why let something as silly as logic get in the way of an opportunity to pound one’s tribal chest & join in a unifying, polarizing group howl?  [All brackets mine– GCM)]

But Kahan’s inference of logical inconsistency cannot be sustained, and thus his speculations as to motive are merely expressions of his own prejudices.

So why didn’t Pew release all the data? Pew’s policy is to release their full data sets a few months after issuing its reports: that’s how I was able to get the full details for the 2009 survey. There may be all sorts of reasons why Pew doesn’t release all its data (most perhaps having to do with the fact that gathering and analyzing such data is most of what they do, and they want first crack at and the chance to publicize their own data before everyone else does), and we won’t be able to check for ourselves to see if Pew made any errors for a few months. But there’s not the slightest hint of error in the released data, and though I don’t know what “tribe” Kahan imagines himself to be a member, if there’s any pounding and howling going on here, it certainly isn’t being done by Pew.

h/t Matthew Cobb