Assessment of Spatio-Temporal Predictors in the Presence of Missing and Heterogeneous Data

TL;DR

This paper presents a residual correlation analysis framework for evaluating the accuracy of spatio-temporal neural models, especially with incomplete and heterogeneous data, by identifying regions where models underperform.

Contribution

It introduces a novel residual correlation method that operates under minimal assumptions to assess and localize model deficiencies in complex spatio-temporal data.

Findings

Effective in detecting underperforming regions in synthetic datasets.

Robust evaluation of deep learning models with missing data.

Applicable to real-world multivariate time series.

Abstract

Deep learning methods achieve remarkable predictive performance in modeling complex, large-scale data. However, assessing the quality of derived models has become increasingly challenging, as more classical statistical assumptions may no longer apply. These difficulties are particularly pronounced for spatio-temporal data, which exhibit dependencies across both space and time and are often characterized by nonlinear dynamics, time variance, and missing observations, hence calling for new accuracy assessment methodologies. This paper introduces a residual correlation analysis framework for assessing the optimality of spatio-temporal relational-enabled neural predictive models, notably in settings with incomplete and heterogeneous data. By leveraging the principle that residual correlation indicates information not captured by the model, enabling the identification and localization of regions in space and time where predictive performance can be improved. A strength of the proposed approach is that it operates under minimal assumptions, allowing also for robust evaluation of deep learning models applied to multivariate time series, even in the presence of missing and heterogeneous data. In detail, the methodology constructs tailored spatio-temporal graphs to encode sparse spatial and temporal dependencies and employs asymptotically distribution-free summary statistics to detect time intervals and spatial regions where the model underperforms. The effectiveness of what proposed is demonstrated through experiments on both synthetic and real-world datasets using state-of-the-art predictive models.