Enhancing LLMs for Governance with Human Oversight: Evaluating and Aligning LLMs on Expert Classification of Climate Misinformation for Detecting False or Misleading Claims about Climate Change

TL;DR

This paper evaluates the ability of large language models to classify climate misinformation, emphasizing the importance of human oversight and expert-annotated data for effective governance and misinformation detection.

Contribution

It demonstrates that fine-tuning GPT-3.5-turbo on expert-annotated datasets achieves expert-level accuracy in climate misinformation classification, outperforming open-source models and existing tools.

Findings

Open-source models underperform proprietary models in climate misinformation classification.

Existing climate-focused tools outperform many proprietary LLMs like GPT-4o.

Fine-tuning GPT-3.5-turbo on expert data achieves expert-level classification accuracy.

Abstract

Climate misinformation is a problem that has the potential to be substantially aggravated by the development of Large Language Models (LLMs). In this study we evaluate the potential for LLMs to be part of the solution for mitigating online dis/misinformation rather than the problem. Employing a public expert annotated dataset and a curated sample of social media content we evaluate the performance of proprietary vs. open source LLMs on climate misinformation classification task, comparing them to existing climate-focused computer-assisted tools and expert assessments. Results show (1) open-source models substantially under-perform in classifying climate misinformation compared to proprietary models, (2) existing climate-focused computer-assisted tools leveraging expert-annotated datasets continues to outperform many of proprietary models, including GPT-4o, and (3) demonstrate the efficacy and generalizability of fine-tuning GPT-3.5-turbo on expert annotated dataset in classifying claims about climate change at the equivalency of climate change experts with over 20 years of experience in climate communication. These findings highlight 1) the importance of incorporating human-oversight, such as incorporating expert-annotated datasets in training LLMs, for governance tasks that require subject-matter expertise like classifying climate misinformation, and 2) the potential for LLMs in facilitating civil society organizations to engage in various governance tasks such as classifying false or misleading claims in domains beyond climate change such as politics and health science.