Consistent Validation for Predictive Methods in Spatial Settings

TL;DR

This paper addresses the challenge of validating spatial prediction methods when validation and test locations differ, proposing a new method that ensures validation accuracy improves with denser data and demonstrating its effectiveness.

Contribution

It introduces a novel validation check for spatial prediction methods and adapts covariate-shift techniques to spatial validation scenarios, with theoretical guarantees.

Findings

Classical validation methods can fail in spatial settings.

The proposed method passes the validation accuracy check.

Empirical results show advantages on simulated and real data.

Abstract

Spatial prediction tasks are key to weather forecasting, studying air pollution impacts, and other scientific endeavors. Determining how much to trust predictions made by statistical or physical methods is essential for the credibility of scientific conclusions. Unfortunately, classical approaches for validation fail to handle mismatch between locations available for validation and (test) locations where we want to make predictions. This mismatch is often not an instance of covariate shift (as commonly formalized) because the validation and test locations are fixed (e.g., on a grid or at select points) rather than i.i.d. from two distributions. In the present work, we formalize a check on validation methods: that they become arbitrarily accurate as validation data becomes arbitrarily dense. We show that classical and covariate-shift methods can fail this check. We propose a method that builds from existing ideas in the covariate-shift literature, but adapts them to the validation data at hand. We prove that our proposal passes our check. And we demonstrate its advantages empirically on simulated and real data.