Inaccuracy through two degrees of separation - Mail mangles science literacy findings by misreporting Fox
- 31 May 2012, 12:00
- Ros Donald
- Mail Online mangles Fox report on Yale study
- Lead researcher says difference between skeptic and
non-skeptics' science literacy is not statistically
significant
According to the
Mail Online yesterday, "Global warming sceptics are
BETTER-informed about science than believers". Pretty
arresting. But a closer look at the article reveals that the Mail's
top line, reporting on research just out, mangles not just the
study, but reporting on the findings by Fox News.
The headline follows a
new study by Yale University psychology researchers about
perceptions of climate change. As we discussed
earlier in the week, the study explores whether levels of
science literacy among ordinary people or their unconscious
tendency to fit their beliefs to those of their social and cultural
groupings are the most accurate indicator of public concern about
climate change.
The study finds finds that contrary to the first theory, the
most scientifically-literate members of the public aren't the most
concerned about climate change. Instead the result fits much more
closely with the second theory - surprisingly, science-savvy people
tend to be even more polarised according to their social groupings
than those less well-informed.
Part of the study aimed to find out how scientifically literate
the subjects were - which was what presumably led the Mail to its
top line:
"57 per cent of climate skeptics are
science literate versus 56 per cent of believers"
Who gets science more?
Now, the Mail's figure doesn't appear in the research paper. So
to get to the bottom of the Mail's rather interesting take on this
finding, it looks as though we need to go to another source
entirely:
Fox News's article on the piece. According to Fox:
"The quiz, containing 22 questions about
both science and statistics, was given to 1,540 representative
Americans. Respondents who were relatively less worried about
global warming got 57 percent of them right, on average, just
barely outscoring those whose who saw global warming as a bigger
threat. They got 56 percent of the questions correct."
We emailed the lead author on the research, Dan Kahan, to find
out where Fox got the information from in the first place. He told
us:
"Who gets science more -- the people who
believe in climate change or those who don't?" was a question
people asked me periodically about the working paper. When they
did, I sent them this graphic:
Kahan explained to us that the graphic shows:
"...the mean or average scores on
the combined science literacy/numeracy scale (a measure of "science
comprehension" measure, essentially) for study subjects whose
responses put them in the top 50% & bottom 50% in "climate
change risk perceptions."
"As you can see, the bottom 50% got 12.6
out of 22 correct. The [top] half got 12.3 correct, on
average."
He adds:
"I sent the journalist this graphic in
response to queries about the working paper. The "56%" & "57%"
figures, I'm guessing, were derived by him by dividing 12.3 &
12.6 by 22, respectively. In any case, they are not results that we
derived from our data."
So it looks as though the Mail messed up Fox News's calculation
based on part of the research findings to say that 57 percent of
those who are doubtful about global warming are "science
literate".
Significant data
All you science literate people will recognise, however, that
there is another aspect to the Mail's error - which it appears to
sort of concede by saying "there's not much in" the difference.
Essentially, any science literacy gap between sceptics and
non-skeptics is pretty meaningless. Kahan says:
"As can be seen, and as I emphasized [to
Fox], this difference isnotstatistically significant. Not even
close."
So - just to sum up. Of the 1,540 people surveyed, those that
were somewhat less concerned about climate change got on average
12.6 general science questions right out of 22. Those that said
they were somewhat more concerned got on average 12.3 questions
right.
This difference is not statistically significant - as Professor
Kahan puts it, "not even close". Fox News's take on this appears to
be at least fairly accurate, saying the study found global warming
skeptics are "
as knowledgeable about science as climate change believers
[...]" - although that's not actually what the study was looking
at, so it would probably be unwise to draw conclusions which are
too definite from it.
So the Mail's conclusion comes, not from garbling original
research, but from misrepresenting a Fox article that has already
skewed its reporting on the results of the Yale study to focus on a
minor aspect - or, in other words, it's an inaccuracy obtained
through two degrees of separation.