Validation of point process predictions with proper scoring rules

TL;DR

This paper introduces flexible proper scoring rules based on summary statistics for evaluating spatial point process forecasts, allowing for calibration assessment and model comparison using simulations and real data applications.

Contribution

It presents a novel class of proper scoring rules that are adaptable, computationally feasible, and more versatile than existing scores for point process prediction evaluation.

Findings

Scoring rules effectively evaluate spatial distribution and clustering.

Simulation studies show sensitivity to forecast aspects.

Applications demonstrate practical usefulness in scientific model selection.

Abstract

We introduce a class of proper scoring rules for evaluating spatial point process forecasts based on summary statistics. These scoring rules rely on Monte-Carlo approximations of expectations and can therefore easily be evaluated for any point process model that can be simulated. In this regard, they are more flexible than the commonly used logarithmic score and other existing proper scores for point process predictions. The scoring rules allow for evaluating the calibration of a model to specific aspects of a point process, such as its spatial distribution or tendency towards clustering. Using simulations we analyze the sensitivity of our scoring rules to different aspects of the forecasts and compare it to the logarithmic score. Applications to earthquake occurrences in northern California, USA and the spatial distribution of Pacific silver firs in Findley Lake Reserve in Washington, USA highlight the usefulness of our scores for scientific model selection.