Moving beyond spatial and random cross-validation in environmental modelling: a call for prediction-domain adaptive evaluation

TL;DR

This paper advocates for prediction-domain adaptive evaluation methods in environmental modeling, addressing the limitations of traditional spatial and random cross-validation techniques.

Contribution

It introduces a new category of adaptive cross-validation methods that better estimate map accuracy across diverse prediction scenarios.

Findings

Random cross-validation works well with randomly distributed training points.

Spatial cross-validation is better suited for extrapolation scenarios.

Adaptive evaluation methods provide more reliable accuracy estimates across different contexts.

Abstract

With the growing application of spatial predictive modeling in ecology, the question of how to appropriately evaluate the resulting maps has gained increasing attention. While there is consensus that map accuracy is ideally estimated using an independent probability sample of the prediction area, there is still no agreement on the most appropriate way to conduct an evaluation for the common case when such a sample is not available. Cross-validation, which involves multiple train-test splits, is commonly applied not only to estimate final model accuracy but also to guide model tuning and selection. Many different spatial and non-spatial approaches to cross-validation have been proposed, and approaches in both groups have faced substantial criticism. It has been shown that random cross-validation methods are suitable when the training points are randomly distributed in the prediction area, while spatial cross-validation is better suited towards extrapolation situations. In practice, however, there is a continuum and most cases are between those two extremes. To address this gap, we advocate for a new category of cross-validation methods to account for this: prediction-domain adaptive evaluation. Methods in this category flexibly adapt to the prediction situation, yielding most reliable estimates of map accuracy across different scenarios. To ground this perspective empirically, we reproduce a simulation study that was used in earlier research and systematically compare different evaluation methods and discuss their purpose.