Navigating the Noise: Bringing Clarity to ML Parameterization Design with O(100) Ensembles

TL;DR

This paper introduces a scalable ensemble-based pipeline to systematically evaluate ML parameterizations for climate models, revealing how offline design choices impact online stability and performance.

Contribution

It presents an end-to-end pipeline for large ensemble simulations, enabling empirical assessment of offline-online performance relationships in ML climate parameterizations.

Findings

Removing dropout and changing loss functions have opposite effects on online stability.

Certain design choices like memory and humidity conversion do not compromise online performance.

Ensemble sizes of around 100 are needed to reliably detect causally relevant online differences.

Abstract

Machine-learning (ML) parameterizations of subgrid processes (here of turbulence, convection, and radiation) may one day replace conventional parameterizations by emulating high-resolution physics without the cost of explicit simulation. However, uncertainty about the relationship between offline and online performance (i.e., when integrated with a large-scale general circulation model (GCM)) hinders their development. Much of this uncertainty stems from limited sampling of the noisy, emergent effects of upstream ML design decisions on downstream online hybrid simulation. Our work rectifies the sampling issue via the construction of a semi-automated, end-to-end pipeline for $\mathcal{O}(100)$ size ensembles of hybrid simulations, revealing important nuances in how systematic reductions in offline error manifest in changes to online error and online stability. For example, removing dropout and switching from a Mean Squared Error (MSE) to a Mean Absolute Error (MAE) loss both reduce offline error, but they have opposite effects on online error and online stability. Other design decisions, like incorporating memory, converting moisture input from specific humidity to relative humidity, using batch normalization, and training on multiple climates do not come with any such compromises. Finally, we show that ensemble sizes of $\mathcal{O}(100)$ may be necessary to reliably detect causally relevant differences online. By enabling rapid online experimentation at scale, we can empirically settle debates regarding subgrid ML parameterization design that would have otherwise remained unresolved in the noise.