Last week I ran a Climate metaquiz, and 123 people responded. The point of a metaquiz is to test how well political groups know the other side, while questions on personal beliefs and knowledge about the climate are secondary. Both the small sample size and potential sampling biases are important caveats to keep in mind here. All that said, Republicans outperformed Democrats on the factual part of the quiz, despite their low self-reported self-confidence. However, Democrats outperformed Republicans on the metaquiz part, with Republicans tending to exaggerate levels of climate change-related handwringing amongst Democrats, as well as their eagerness to exaggerate the facts in the name of behavior change.

Sanity checking, the self-reported data matches stereotypes. 60% of Republican test takers believe climate change is less serious than the scientific consensus, compared to 6% of Democrats. Democrat test takers were far more concerned (2.8 on a 5-point likert scale) about climate change affecting them personally and future generations than Republicans (1.8). Republicans were very skeptical about water level increases by 2100, predicting an average of less than 2 feet rise, while democrats were more concerned, predicting on average more than 3 feet.

Actual vs. predicted results

Since this was a metaquiz, the focus was not on the participants' own beliefs and knowledge of the climate, but their predictions about the other side. The reason for the first sections was only to establish a baseline. Here are the aggregated results. I calculate percentage error using this simple formula: % error = (actual - predicted) / actual:

Interesting findings

In aggregate, participants underestimated how well the other side did on the quiz (total 10 points). Republicans predicted 4.84 (actual 5.9) for Democrats, while Democrats predicted 4.95 (actual 6.39) for Republicans. About 20% of all respondents thought they did better than the other side, as many thought that they did worse, while the rest weren't sure. One exception here are Republicans, who exhibited false modesty: 40% of them thought that the Democrats would perform better.

Exaggerating facts: Republicans thought Democrats would be much more comfortable exaggerating scientific facts to convince others of environmentally beneficial behavior change (predicted 2.9) more so than Democrats declared (reported 1.6). On the flip side, democrats didn't think the Republicans would be comfortable exaggerating scientific facts for the environment (predicted 1.4), and they were right (reported: 1.2). It was pointed out to me that the way I phrased the question was quite leading: "It's okay to exaggerate scientific facts in order to convince others to behave in a more environmentally friendly way." but I'm not sure which way this would bias quiz and meta-quiz takers.

Impact of climate change: Democrats accurately predicted Republican lack of concern about climate change affecting them personally (reported: 1.8, predicted: 1.7) and future generations (reported: 2.7, predicted: 2.3). Republicans predicted Democrats would be slightly more concerned about the climate than they were both personally (reported: 2.8, predicted: 3.5) and for future generations (reported: 4.2, predicted: 4.5). The same trend was repeated when asked about water level rise by 2100. Democrats correctly predicted that Republicans would be conservative about future water level rises (reported: 1.8 ft, predicted: 1.8 ft), while Republicans were way off about what Democrats would think (reported: 3.1 ft, predicted: 4.7 ft). This asymetry is intriguing.

I must again caveat all of this with the fact that 123 quiz takers does not constitute a statistically significant sample, not to mention selection biases that come from posting the quiz on twitter and some rationalist forums. Still, it's interesting to see the consistency with which Republicans tended to exaggerate Democratic positions more than vice-versa. One explanation is that Paul Krugman was right when he said "A liberal can talk coherently about what the conservative view is because people like me actually do listen." Another explanation is that this disparity is due to the unique dynamics of climate change, which is a much more important issue for Democrats than for Republicans. It would be interesting to do more metaquizzes on other topics that are more balanced in perceived importantness. Perhaps the metaquiz could be framed as a "competition in understanding" between the two sides.

I find the metaquiz format to be an interesting one, serving a purpose similar to the Ideological Turing Test, but requiring less effort of everyone involved (ie. essay reading or writing). Thanks again to the survey takers. If you left your email and are curious about how well you did on the metaquiz part, let me know and I'll send you your personal results.

As always, please send me feedback on the metaquiz concept via twitter or email.