Spatially-informed transformers: Injecting geostatistical covariance biases into self-attention for spatio-temporal forecasting

TL;DR

This paper introduces a hybrid transformer architecture that incorporates geostatistical covariance biases into self-attention, improving spatio-temporal forecasting by combining probabilistic rigor with deep learning flexibility.

Contribution

It proposes a spatially-informed transformer that injects geostatistical inductive bias into self-attention via a learnable covariance kernel, bridging geostatistics and deep learning.

Findings

Outperforms state-of-the-art graph neural networks on benchmarks.

Recovers true spatial decay parameters end-to-end.

Provides well-calibrated probabilistic forecasts.

Abstract

The modeling of high-dimensional spatio-temporal processes presents a fundamental dichotomy between the probabilistic rigor of classical geostatistics and the flexible, high-capacity representations of deep learning. While Gaussian processes offer theoretical consistency and exact uncertainty quantification, their prohibitive computational scaling renders them impractical for massive sensor networks. Conversely, modern transformer architectures excel at sequence modeling but inherently lack a geometric inductive bias, treating spatial sensors as permutation-invariant tokens without a native understanding of distance. In this work, we propose a spatially-informed transformer, a hybrid architecture that injects a geostatistical inductive bias directly into the self-attention mechanism via a learnable covariance kernel. By formally decomposing the attention structure into a stationary physical prior and a non-stationary data-driven residual, we impose a soft topological constraint that favors spatially proximal interactions while retaining the capacity to model complex dynamics. We demonstrate the phenomenon of ``Deep Variography'', where the network successfully recovers the true spatial decay parameters of the underlying process end-to-end via backpropagation. Extensive experiments on synthetic Gaussian random fields and real-world traffic benchmarks confirm that our method outperforms state-of-the-art graph neural networks. Furthermore, rigorous statistical validation confirms that the proposed method delivers not only superior predictive accuracy but also well-calibrated probabilistic forecasts, effectively bridging the gap between physics-aware modeling and data-driven learning.