(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models

TL;DR

Mosaic is a novel probabilistic weather forecasting model that preserves spectral fidelity and outperforms finer-resolution models using mesh-aligned block-sparse attention, enabling efficient, high-quality ensemble predictions.

Contribution

The paper introduces Mosaic, a spectral fidelity-preserving probabilistic weather model utilizing mesh-aligned block-sparse attention for efficient long-range dependency capture.

Findings

Mosaic matches or exceeds finer-resolution models on key variables.

Produces well-calibrated ensembles with near-perfect spectral alignment.

Forecasts are computed in under 12 seconds on a single GPU.

Abstract

We introduce Mosaic, a probabilistic weather forecasting model that addresses three failure modes of spectral degradation in ML-based weather prediction: spectral damping (statistical), high-frequency aliasing (architectural), and residual high-frequency leakage (parametric). Mosaic generates ensemble members through learned functional perturbations and operates on native-resolution grids via mesh-aligned block-sparse attention, a hardware-aligned mechanism that captures long-range dependencies at linear cost by sharing keys and values across spatially adjacent queries. At 1.5{\deg} resolution with 214M parameters, Mosaic matches or outperforms models trained on 6$\times$ finer resolution on key variables and achieves state-of-the-art results among 1.5{\deg} models, producing well-calibrated ensembles whose individual members exhibit near-perfect spectral alignment across all resolved frequencies. A 24-member, 10-day forecast takes under 12s on a single H100~GPU. Code is available at https://github.com/maxxxzdn/mosaic.