Field-Space Attention for Structure-Preserving Earth System Transformers

TL;DR

This paper introduces Field-Space attention, a novel Earth system Transformer mechanism that operates directly on physical fields to enhance interpretability, stability, and physical consistency in Earth system modeling.

Contribution

It presents a new attention mechanism that computes in the physical domain, preserving geometric structure and enabling better integration of multiscale Earth system data.

Findings

Faster and more stable convergence than standard Vision Transformers.

Requires fewer parameters while maintaining high fidelity.

Enables embedding of physical priors directly into the model.

Abstract

Accurate and physically consistent modeling of Earth system dynamics requires machine-learning architectures that operate directly on continuous geophysical fields and preserve their underlying geometric structure. Here we introduce Field-Space attention, a mechanism for Earth system Transformers that computes attention in the physical domain rather than in a learned latent space. By maintaining all intermediate representations as continuous fields on the sphere, the architecture enables interpretable internal states and facilitates the enforcement of scientific constraints. The model employs a fixed, non-learned multiscale decomposition and learns structure-preserving deformations of the input field, allowing coherent integration of coarse and fine-scale information while avoiding the optimization instabilities characteristic of standard single-scale Vision Transformers. Applied to global temperature super-resolution on a HEALPix grid, Field-Space Transformers converge more rapidly and stably than conventional Vision Transformers and U-Net baselines, while requiring substantially fewer parameters. The explicit preservation of field structure throughout the network allows physical and statistical priors to be embedded directly into the architecture, yielding improved fidelity and reliability in data-driven Earth system modeling. These results position Field-Space Attention as a compact, interpretable, and physically grounded building block for next-generation Earth system prediction and generative modeling frameworks.