Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts

TL;DR

This paper evaluates the out-of-distribution robustness of climate emulators, including a foundation model, under no-analog climate scenarios caused by external forcing changes, highlighting the importance of scenario-aware training.

Contribution

It benchmarks the OOD robustness of three climate models, revealing the trade-offs between accuracy and stability under distribution shifts and emphasizing the need for scenario-aware training.

Findings

ClimaX achieves the lowest absolute error but shows higher relative performance changes under shifts.

Precipitation errors increase by up to 8.44% under extreme forcing scenarios.

High-capacity models are sensitive to external forcing trajectories when trained only on historical data.

Abstract

The accelerating pace of climate change introduces profound non-stationarities that challenge the ability of Machine Learning based climate emulators to generalize beyond their training distributions. While these emulators offer computationally efficient alternatives to traditional Earth System Models, their reliability remains a potential bottleneck under "no-analog" future climate states, which we define here as regimes where external forcing drives the system into conditions outside the empirical range of the historical training data. A fundamental challenge in evaluating this reliability is data contamination; because many models are trained on simulations that already encompass future scenarios, true out-of-distribution (OOD) performance is often masked. To address this, we benchmark the OOD robustness of three state-of-the-art architectures: U-Net, ConvLSTM, and the ClimaX foundation model specifically restricted to a historical-only training regime (1850-2014). We evaluate these models using two complementary strategies: (i) temporal extrapolation to the recent climate (2015-2023) and (ii) cross-scenario forcing shifts across divergent emission pathways. Our analysis within this experimental setup reveals an accuracy vs. stability trade-off: while the ClimaX foundation model achieves the lowest absolute error, it exhibits higher relative performance changes under distribution shifts, with precipitation errors increasing by up to 8.44% under extreme forcing scenarios. These findings suggest that when restricted to historical training dynamics, even high-capacity foundation models are sensitive to external forcing trajectories. Our results underscore the necessity of scenario-aware training and rigorous OOD evaluation protocols to ensure the robustness of climate emulators under a changing climate.