The impact of internal variability on benchmarking deep learning climate emulators

TL;DR

This paper examines how internal variability affects the performance of deep learning climate emulators, revealing that simpler linear models can outperform complex models on certain variables due to overfitting to noise.

Contribution

It demonstrates that increasing simulation data improves emulator accuracy and highlights the impact of internal variability on deep learning models in climate emulation.

Findings

Linear emulator outperforms deep learning model on temperature and precipitation with limited data.

Deep learning models tend to overfit internal variability noise, reducing accuracy.

More simulation data improves emulator performance, especially for noisy variables.

Abstract

Full-complexity Earth system models (ESMs) are computationally very expensive, limiting their use in exploring the climate outcomes of multiple emission pathways. More efficient emulators that approximate ESMs can directly map emissions onto climate outcomes, and benchmarks are being used to evaluate their accuracy on standardized tasks and datasets. We investigate a popular benchmark in data-driven climate emulation, ClimateBench, on which deep learning-based emulators are currently achieving the best performance. We compare these deep learning emulators with a linear regression-based emulator, akin to pattern scaling, and show that it outperforms the incumbent 100M-parameter deep learning foundation model, ClimaX, on 3 out of 4 regionally-resolved climate variables, notably surface temperature and precipitation. While emulating surface temperature is expected to be predominantly linear, this result is surprising for emulating precipitation. Precipitation is a much more noisy variable, and we show that deep learning emulators can overfit to internal variability noise at low frequencies, degrading their performance in comparison to a linear emulator. We address the issue of overfitting by increasing the number of climate simulations per emission pathway (from 3 to 50) and updating the benchmark targets with the respective ensemble averages from the MPI-ESM1.2-LR model. Using the new targets, we show that linear pattern scaling continues to be more accurate on temperature, but can be outperformed by a deep learning-based technique for emulating precipitation. We publish our code and data at github.com/blutjens/climate-emulator.