Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification

TL;DR

This paper evaluates deep ensemble neural networks for quantifying uncertainty in climate predictions, specifically for ENSO, revealing that epistemic uncertainty reliably indicates prediction errors while aleatoric uncertainty becomes less dependable under climate change.

Contribution

It demonstrates the effectiveness of deep ensembles in distinguishing epistemic and aleatoric uncertainties in climate models, especially under distributional shifts caused by climate change.

Findings

Epistemic uncertainty signals predictive error growth during climate shifts.

Aleatoric uncertainty becomes less reliable under climate change.

Ensemble performance improves with increased epistemic uncertainty.

Abstract

Faithful uncertainty quantification (UQ) is paramount in high stakes climate prediction. Deep ensembles, or ensembles of probabilistic neural networks, are state of the art for UQ in machine learning (ML) and are growing increasingly popular for weather and climate prediction. However, detailed analyses of the mechanisms, strengths, and limitations of ensembles in these complex problem settings are lacking. We take a step towards filling this gap by deploying deep ensembles for predictability analysis of the El-Ni\~no Southern Oscillation (ENSO) in the Community Earth System Model 2 Large Ensemble (CESM2-LE). Principally, we show that epistemic uncertainty, modeled by ensemble disagreement, robustly signals predictive error growth associated with shifts in the distributions of monthly sea-surface temperature (SST), ocean heat content (OHC), and zonal surface wind stress ($\tau_x$) anomalies under a climate change scenario. Conversely, we find that aleatoric uncertainty, which remains a popular measure of model confidence, becomes less reliable and behaves counterintuitively under climate-change-induced distributional shift. We highlight that, because ensemble performance improvement relative to the expected single model scales with epistemic uncertainty, ensemble improvement increases with distributional shift from climate change. This work demonstrates the utility of deep ensembles for modeling aleatoric and epistemic uncertainty in ML climate prediction, as well as the growing importance of robustly quantifying these two forms of uncertainty under anthropogenic warming.