GeOT: A spatially explicit framework for evaluating spatio-temporal predictions

TL;DR

This paper introduces GeOT, a novel spatial evaluation framework using Optimal Transport to measure and improve the spatial accuracy of spatio-temporal prediction models, addressing limitations of traditional point-wise metrics.

Contribution

The paper proposes GeOT, a spatially explicit evaluation and training framework based on Optimal Transport, to better assess and enhance the spatial accuracy of predictions in GeoAI.

Findings

OT captures spatial costs more accurately than existing metrics.

Spatial error distribution relates to real-world costs in applications.

OT-based training reduces spatial prediction errors with minimal impact on non-spatial metrics.

Abstract

When predicting observations across space and time, the spatial layout of errors impacts a model's real-world utility. For instance, in bike sharing demand prediction, error patterns translate to relocation costs. However, commonly used error metrics in GeoAI evaluate predictions point-wise, neglecting effects such as spatial heterogeneity, autocorrelation, and the Modifiable Areal Unit Problem. We put forward Optimal Transport (OT) as a spatial evaluation metric and loss function. The proposed framework, called GeOT, assesses the performance of prediction models by quantifying the transport costs associated with their prediction errors. Through experiments on real and synthetic data, we demonstrate that 1) the spatial distribution of prediction errors relates to real-world costs in many applications, 2) OT captures these spatial costs more accurately than existing metrics, and 3) OT enhances comparability across spatial and temporal scales. Finally, we advocate for leveraging OT as a loss function in neural networks to improve the spatial accuracy of predictions. Experiments with bike sharing, charging station, and traffic datasets show that spatial costs are significantly reduced with only marginal changes to non-spatial error metrics. Thus, this approach not only offers a spatially explicit tool for model evaluation and selection, but also integrates spatial considerations into model training. All code is available at https://github.com/mie-lab/geospatialOT.