Generative Debunking of Climate Misinformation

TL;DR

This paper develops large language models that generate structured climate misinformation debunkings, combining open and proprietary models with prompting strategies, and releases a dataset and demo for future research.

Contribution

It introduces a novel LLM prompting framework for automatic climate misinformation debunking using the fact-myth-fallacy-fact structure, with a new dataset and system release.

Findings

GPT-4 and Mixtral perform well with structured prompts

Structured prompting improves debunking quality

Challenges remain in debunking generation and evaluation

Abstract

Misinformation about climate change causes numerous negative impacts, necessitating corrective responses. Psychological research has offered various strategies for reducing the influence of climate misinformation, such as the fact-myth-fallacy-fact-structure. However, practically implementing corrective interventions at scale represents a challenge. Automatic detection and correction of misinformation offers a solution to the misinformation problem. This study documents the development of large language models that accept as input a climate myth and produce a debunking that adheres to the fact-myth-fallacy-fact (``truth sandwich'') structure, by incorporating contrarian claim classification and fallacy detection into an LLM prompting framework. We combine open (Mixtral, Palm2) and proprietary (GPT-4) LLMs with prompting strategies of varying complexity. Experiments reveal promising performance of GPT-4 and Mixtral if combined with structured prompts. We identify specific challenges of debunking generation and human evaluation, and map out avenues for future work. We release a dataset of high-quality truth-sandwich debunkings, source code and a demo of the debunking system.