Evaluating the Generalization Ability of Spatiotemporal Model in Urban Scenario

TL;DR

This paper introduces a new benchmark for evaluating the generalization of spatiotemporal models in urban scenarios, revealing significant performance drops out-of-distribution and exploring dropout as a mitigation strategy.

Contribution

The paper proposes the ST-OOD benchmark for assessing spatiotemporal model generalization in urban environments and evaluates state-of-the-art models' robustness to distribution shifts.

Findings

Models perform poorly out-of-distribution, often worse than simple MLPs.

Dropout can improve out-of-distribution generalization with minimal in-distribution impact.

Current models tend to overfit training data, reducing robustness.

Abstract

Spatiotemporal neural networks have shown great promise in urban scenarios by effectively capturing temporal and spatial correlations. However, urban environments are constantly evolving, and current model evaluations are often limited to traffic scenarios and use data mainly collected only a few weeks after training period to evaluate model performance. The generalization ability of these models remains largely unexplored. To address this, we propose a Spatiotemporal Out-of-Distribution (ST-OOD) benchmark, which comprises six urban scenario: bike-sharing, 311 services, pedestrian counts, traffic speed, traffic flow, ride-hailing demand, and bike-sharing, each with in-distribution (same year) and out-of-distribution (next years) settings. We extensively evaluate state-of-the-art spatiotemporal models and find that their performance degrades significantly in out-of-distribution settings, with most models performing even worse than a simple Multi-Layer Perceptron (MLP). Our findings suggest that current leading methods tend to over-rely on parameters to overfit training data, which may lead to good performance on in-distribution data but often results in poor generalization. We also investigated whether dropout could mitigate the negative effects of overfitting. Our results showed that a slight dropout rate could significantly improve generalization performance on most datasets, with minimal impact on in-distribution performance. However, balancing in-distribution and out-of-distribution performance remains a challenging problem. We hope that the proposed benchmark will encourage further research on this critical issue.