Inaccuracy through two degrees of separation - Mail mangles science literacy findings by misreporting Fox

  • 31 May 2012, 12:00
  • Ros Donald
  • Mail Online mangles Fox report on Yale study
  • Lead researcher says difference between skeptic and non-skeptics' science literacy is not statistically significant

According to the Mail Online yesterday, "Global warming sceptics are BETTER-informed about science than believers".  Pretty arresting. But a closer look at the article reveals that the Mail's top line, reporting on research just out, mangles not just the study, but reporting on the findings by Fox News.

The headline follows a new study by Yale University psychology researchers about perceptions of climate change. As we discussed earlier in the week, the study explores whether levels of science literacy among ordinary people or their unconscious tendency to fit their beliefs to those of their social and cultural groupings are the most accurate indicator of public concern about climate change.

The study finds finds that contrary to the first theory, the most scientifically-literate members of the public aren't the most concerned about climate change. Instead the result fits much more closely with the second theory - surprisingly, science-savvy people tend to be even more polarised according to their social groupings than those less well-informed.

Part of the study aimed to find out how scientifically literate the subjects were - which was what presumably led the Mail to its top line:

"57 per cent of climate skeptics are science literate versus 56 per cent of believers"

Who gets science more?

Now, the Mail's figure doesn't appear in the research paper. So to get to the bottom of the Mail's rather interesting take on this finding, it looks as though we need to go to another source entirely: Fox News's article on the piece. According to Fox:

"The quiz, containing 22 questions about both science and statistics, was given to 1,540 representative Americans. Respondents who were relatively less worried about global warming got 57 percent of them right, on average, just barely outscoring those whose who saw global warming as a bigger threat. They got 56 percent of the questions correct."

We emailed the lead author on the research, Dan Kahan, to find out where Fox got the information from in the first place. He told us:

"Who gets science more -- the people who believe in climate change or those who don't?" was a question people asked me periodically about the working paper. When they did, I sent them this graphic:

Science Literacy

Kahan explained to us that the graphic shows:

 "...the mean or average scores on the combined science literacy/numeracy scale (a measure of "science comprehension" measure, essentially) for study subjects whose responses put them in the top 50% & bottom 50% in "climate change risk perceptions."

"As you can see, the bottom 50% got 12.6 out of 22 correct. The [top] half got 12.3 correct, on average."

He adds:

"I sent the journalist this graphic in response to queries about the working paper. The "56%" & "57%" figures, I'm guessing, were derived by him by dividing 12.3 & 12.6 by 22, respectively. In any case, they are not results that we derived from our data."

So it looks as though the Mail messed up Fox News's calculation based on part of the research findings to say that 57 percent of those who are doubtful about global warming are "science literate".

Significant data

All you science literate people will recognise, however, that there is another aspect to the Mail's error - which it appears to sort of concede by saying "there's not much in" the difference. Essentially, any science literacy gap between sceptics and non-skeptics is pretty meaningless. Kahan says:

"As can be seen, and as I emphasized [to Fox], this difference isnotstatistically significant. Not even close."

So - just to sum up. Of the 1,540 people surveyed, those that were somewhat less concerned about climate change got on average 12.6 general science questions right out of 22. Those that said they were somewhat more concerned got on average 12.3 questions right.

This difference is not statistically significant - as Professor Kahan puts it, "not even close". Fox News's take on this appears to be at least fairly accurate, saying the study found global warming skeptics are " as knowledgeable about science as climate change believers [...]" - although that's not actually what the study was looking at, so it would probably be unwise to draw conclusions which are too definite from it.  

So the Mail's conclusion comes, not from garbling original research, but from misrepresenting a Fox article that has already skewed its reporting on the results of the Yale study to focus on a minor aspect - or, in other words, it's an inaccuracy obtained through two degrees of separation.

Email Share to Facebook Stumble It
blog comments powered by Disqus