ExEBench: Benchmarking Foundation Models on Extreme Earth Events

TL;DR

ExEBench is a comprehensive benchmark dataset designed to evaluate foundation models' ability to handle diverse extreme earth events, aiming to improve disaster management and understanding of climate impacts.

Contribution

The paper introduces ExEBench, a new dataset and evaluation platform for testing foundation models on multiple extreme earth event categories with real-world operational tasks.

Findings

Assesses FM generalizability across diverse extreme event tasks.

Provides a platform for developing novel ML methods for disaster management.

Facilitates analysis of cascading effects of extreme events.

Abstract

Our planet is facing increasingly frequent extreme events, which pose major risks to human lives and ecosystems. Recent advances in machine learning (ML), especially with foundation models (FMs) trained on extensive datasets, excel in extracting features and show promise in disaster management. Nevertheless, these models often inherit biases from training data, challenging their performance over extreme values. To explore the reliability of FM in the context of extreme events, we introduce \textbf{ExE}Bench (\textbf{Ex}treme \textbf{E}arth Benchmark), a collection of seven extreme event categories across floods, wildfires, storms, tropical cyclones, extreme precipitation, heatwaves, and cold waves. The dataset features global coverage, varying data volumes, and diverse data sources with different spatial, temporal, and spectral characteristics. To broaden the real-world impact of FMs, we include multiple challenging ML tasks that are closely aligned with operational needs in extreme events detection, monitoring, and forecasting. ExEBench aims to (1) assess FM generalizability across diverse, high-impact tasks and domains, (2) promote the development of novel ML methods that benefit disaster management, and (3) offer a platform for analyzing the interactions and cascading effects of extreme events to advance our understanding of Earth system, especially under the climate change expected in the decades to come. The dataset and code are public https://github.com/zhaoshan2/EarthExtreme-Bench.