WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models

TL;DR

This paper introduces WXImpactBench, a novel benchmark dataset designed to evaluate large language models' ability to understand disruptive weather impacts, aiding climate change adaptation efforts.

Contribution

It presents the first comprehensive benchmark and dataset for assessing LLMs on disruptive weather impact understanding, including a four-stage data construction pipeline.

Findings

LLMs face significant challenges in understanding disruptive weather impacts.

The benchmark reveals current limitations of LLMs in climate-related tasks.

The dataset and evaluation framework are publicly available for further research.

Abstract

Climate change adaptation requires the understanding of disruptive weather impacts on society, where large language models (LLMs) might be applicable. However, their effectiveness is under-explored due to the difficulty of high-quality corpus collection and the lack of available benchmarks. The climate-related events stored in regional newspapers record how communities adapted and recovered from disasters. However, the processing of the original corpus is non-trivial. In this study, we first develop a disruptive weather impact dataset with a four-stage well-crafted construction pipeline. Then, we propose WXImpactBench, the first benchmark for evaluating the capacity of LLMs on disruptive weather impacts. The benchmark involves two evaluation tasks, multi-label classification and ranking-based question answering. Extensive experiments on evaluating a set of LLMs provide first-hand analysis of the challenges in developing disruptive weather impact understanding and climate change adaptation systems. The constructed dataset and the code for the evaluation framework are available to help society protect against vulnerabilities from disasters.