Mitigation of Spatial Nonstationarity with Vision Transformers

TL;DR

This paper explores how vision transformers can effectively mitigate spatial nonstationarity in geospatial data, outperforming CNNs and providing a new deep learning approach for complex spatial modeling tasks.

Contribution

It introduces the use of self-attention (vision transformer) models to address spatial nonstationarity in geospatial data, demonstrating superior performance over traditional deep learning methods.

Findings

Vision transformers reduce prediction errors to as low as 10%.

Self-attention models outperform CNNs in nonstationary spatial data modeling.

The approach establishes best practices for large-scale spatial relationship modeling.

Abstract

Spatial nonstationarity, the location variance of features' statistical distributions, is ubiquitous in many natural settings. For example, in geological reservoirs rock matrix porosity varies vertically due to geomechanical compaction trends, in mineral deposits grades vary due to sedimentation and concentration processes, in hydrology rainfall varies due to the atmosphere and topography interactions, and in metallurgy crystalline structures vary due to differential cooling. Conventional geostatistical modeling workflows rely on the assumption of stationarity to be able to model spatial features for the geostatistical inference. Nevertheless, this is often not a realistic assumption when dealing with nonstationary spatial data and this has motivated a variety of nonstationary spatial modeling workflows such as trend and residual decomposition, cosimulation with secondary features, and spatial segmentation and independent modeling over stationary subdomains. The advent of deep learning technologies has enabled new workflows for modeling spatial relationships. However, there is a paucity of demonstrated best practice and general guidance on mitigation of spatial nonstationarity with deep learning in the geospatial context. We demonstrate the impact of two common types of geostatistical spatial nonstationarity on deep learning model prediction performance and propose the mitigation of such impacts using self-attention (vision transformer) models. We demonstrate the utility of vision transformers for the mitigation of nonstationarity with relative errors as low as 10%, exceeding the performance of alternative deep learning methods such as convolutional neural networks. We establish best practice by demonstrating the ability of self-attention networks for modeling large-scale spatial relationships in the presence of commonly observed geospatial nonstationarity.