Self-Attentive Ensemble Transformer: Representing Ensemble Interactions in Neural Networks for Earth System Models

TL;DR

This paper introduces a novel neural network approach called the self-attentive ensemble transformer, which improves calibration and information extraction from ensemble Earth system model forecasts by representing interactions between ensemble members.

Contribution

It presents a new member-by-member neural network method that incorporates self-attention to model ensemble interactions for post-processing Earth system model forecasts.

Findings

The ensemble transformer effectively calibrates ensemble spread.

It extracts additional information from ensemble data.

Outputs spatially-coherent, multivariate ensemble members.

Abstract

Ensemble data from Earth system models has to be calibrated and post-processed. I propose a novel member-by-member post-processing approach with neural networks. I bridge ideas from ensemble data assimilation with self-attention, resulting into the self-attentive ensemble transformer. Here, interactions between ensemble members are represented as additive and dynamic self-attentive part. As proof-of-concept, I regress global ECMWF ensemble forecasts to 2-metre-temperature fields from the ERA5 reanalysis. I demonstrate that the ensemble transformer can calibrate the ensemble spread and extract additional information from the ensemble. As it is a member-by-member approach, the ensemble transformer directly outputs multivariate and spatially-coherent ensemble members. Therefore, self-attention and the transformer technique can be a missing piece for a non-parametric post-processing of ensemble data with neural networks.