Prediction & Model Evaluation for Space-Time Data

TL;DR

This paper investigates evaluation metrics for space-time prediction models, emphasizing the importance of appropriate cross-validation methods like LOLO CV for spatial interpolation, especially in dependent data scenarios such as wildfire air pollution data.

Contribution

It formalizes the true prediction error for spatial interpolation and evaluates cross-validation strategies, recommending LOLO CV as the most suitable method for dependent space-time data.

Findings

Location-based CV accurately estimates spatial interpolation error.

LOLO CV is recommended for dependent space-time data.

Common bias-variance assumptions do not hold for dependent data.

Abstract

Evaluation metrics for prediction error, model selection and model averaging on space-time data are understudied and poorly understood. The absence of independent replication makes prediction ambiguous as a concept and renders evaluation procedures developed for independent data inappropriate for most space-time prediction problems. Motivated by air pollution data collected during California wildfires in 2008, this manuscript attempts a formalization of the true prediction error associated with spatial interpolation. We investigate a variety of cross-validation (CV) procedures employing both simulations and case studies to provide insight into the nature of the estimand targeted by alternative data partition strategies. Consistent with recent best practice, we find that location-based cross-validation is appropriate for estimating spatial interpolation error as in our analysis of the California wildfire data. Interestingly, commonly held notions of bias-variance trade-off of CV fold size do not trivially apply to dependent data, and we recommend leave-one-location-out (LOLO) CV as the preferred prediction error metric for spatial interpolation.