How confident should we be..? A psychologist answers our question about his research
- 30 May 2012, 15:22
- Freya Roberts
How confident can we be about the results of scientific studies?
It's not a question the media often ask when writing about climate
change. Research which hasn't been even finished yet has for
example been given as conclusive proof that climate change
isn't happening, or, alternatively, is causing
us to shrink. The author of a
recent study, which tests two competing theories to explain how
people perceive risk about climate change, discusses the issue of
confidence on his
blog, in response to a question we sent him.
As we
wrote on Monday, the paper concludes that social groupings are
a much better measure for predicting whether or not people will
express concern about climate change than an individual's level of
scientific literacy.
We got in touch with lead author, Dan Kahan, to ask about one of
the effects he noticed - that with increasing science literacy,
people actually became more divided on the risks posed by climate
change. We asked:
"From the figure (Fig.2) [the polarising
effect] appears to be quite subtle, albeit in the opposite
direction to that which was predicted by the SCT thesis. It would
be great if you could identify to what extent/how confident we can
be to say that increasing numeracy and literacy polarises risk
perception about climate change"
The purpose of the research
First, he describes the theories he wanted to test in the
study:
"As you know, our study investigated two
hypotheses: the science comprehension thesis (SCT), which
attributes public conflict over climate change to deficits in
science comprehension; and the cultural cognition thesis (CCT),
which asserts that conflict over climate change is a consequence of
the unconscious tendency of individuals to fit the beliefs about
risk to positions that dominate in their group, and which in its
strongest form would say that this tendency will be reinforced or
magnified by grater science comprehension, which can be used to
promote such fitting."
CCT theory identifies two distinct groups within the public:
egalitarian communitarians who perceive the risk to be higher, and
hierarchical individualists who perceive the risk to be lower - so
according to this theory social grouping will be the most
significant indicator of how concerned people are about climate
change. In contrast, SCT theory suggest individuals, regardless of
their social grouping, should be more concerned about climate
change risk the more science literate they are.
If SCT theory is correct, then as science literacy increases,
both skeptics and non-skeptics should perceive the risk to be
higher. The gap in risk perception between the two groups should
not get larger. But the study shows the gap does get larger:
people's views become more polarised, and that can be explained by
CCT:
"The study is consistent with CCT and
furnishes modest evidence that CCT in its strongest form is
correct. That position would predict that cultural polarization
will be greater among individuals with the greatest science
comprehension. The results fit that hypothesis."
Confidence in conclusions
Given that the results give only "modest evidence" for CCT, we
wanted to know how confident we can be in drawing these
conclusions. To answer our question, Dan says there is:
"only modest evidence [for CCT thesis]
mainly because of the design of the study. It's observational
--correlational -- only. Observed correlations that fit a
hypothesis are strong in proportion to which they rule out other
explanations. Maybe something else is going on that causes both
increased science comprehension & increased polarization in
certain people."
There are also other reasons to be measured when making claims
based on one study:
"All study results are provisional.
That's in the nature of science. Valid studies give you more
evidence than you otherwise would have had to believe something.
They never "settle" the issue."
"It is never sensible (it is a
misundersanding of the nature of empirical proof) to say, "this
study proves this" or "this study doesn't necessary prove
that ..." etc."
Building on the research
But the research doesn't stop here. Professor Kahan says further
experiments can build on the evidence for the CCT thesis and help
to rule out other explanations:
"The only way to tell [if something else
is going on] is through (well designed) experiments. We are
conducting some now."
"...if the source of the effect is CCT,
it should be easy to produce much more dramatic effects through
properly designed experiments. So rather than try to extract more
information from the effect size about how confident or not to be
in the strong CCT position, it makes sense to do experiments. Again
that's what we are now doing."
Understanding the design of this experiment helps us to know
what the study can and can't tell us. By comparing scientific
literacy and numeracy to risk perception, the research can only
identify trends in how risk perception changes. It can't identify a
causal link between the two.
"Nothing in our study suggests that
making people more science literate or numerate causes
polarization. If CCT is correct, there is something about
climate change (and certain other issues) that makes people try to
maximize the fit between their beliefs and positions that
predominate within their groups, which themselves are impelled into
opposing stances on certain facts. That thing is the cause in the
practical, normative sense."
These findings, combined with those from future research can
greatly add to our understanding on what divides public perception
of climate change risk. This study alone can't explicitly 'prove'
either theory, it can simply add more evidence to our understanding
of risk perception. That too is in the nature of science.
You can read Professor Dan Kahan's blog, which also contains
references to other papers, in full
on the Cultural Cognition blog .