Neural Compression of Atmospheric States

TL;DR

This paper introduces a neural network-based compression method for atmospheric data, achieving over 1000x reduction in data size while preserving key features and enabling rapid processing of global atmospheric states.

Contribution

It adapts neural compression techniques to spherical atmospheric data using HEALPix projection, demonstrating high compression ratios with faithful reconstruction of critical weather phenomena.

Findings

Achieved over 1000x compression ratios.

Reconstructed extreme weather events accurately.

Maintained spectral power distribution across scales.

Abstract

Atmospheric states derived from reanalysis comprise a substantial portion of weather and climate simulation outputs. Many stakeholders -- such as researchers, policy makers, and insurers -- use this data to better understand the earth system and guide policy decisions. Atmospheric states have also received increased interest as machine learning approaches to weather prediction have shown promising results. A key issue for all audiences is that dense time series of these high-dimensional states comprise an enormous amount of data, precluding all but the most well resourced groups from accessing and using historical data and future projections. To address this problem, we propose a method for compressing atmospheric states using methods from the neural network literature, adapting spherical data to processing by conventional neural architectures through the use of the area-preserving HEALPix projection. We investigate two model classes for building neural compressors: the hyperprior model from the neural image compression literature and recent vector-quantised models. We show that both families of models satisfy the desiderata of small average error, a small number of high-error reconstructed pixels, faithful reproduction of extreme events such as hurricanes and heatwaves, preservation of the spectral power distribution across spatial scales. We demonstrate compression ratios in excess of 1000x, with compression and decompression at a rate of approximately one second per global atmospheric state.