OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

TL;DR

OpenSTL introduces a comprehensive benchmark for spatio-temporal predictive learning, systematically evaluating diverse models across multiple datasets to understand their performance and efficiency, highlighting the competitiveness of recurrent-free approaches.

Contribution

It provides a standardized, extensible framework for evaluating spatio-temporal models and offers new insights into the effectiveness of recurrent-free architectures.

Findings

Recurrent-free models balance efficiency and performance better than recurrent models.

Model architecture and dataset properties significantly influence predictive learning performance.

Extending MetaFormers enhances recurrent-free spatial-temporal predictive learning.

Abstract

Spatio-temporal predictive learning is a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsupervised manner. Despite remarkable progress in recent years, a lack of systematic understanding persists due to the diverse settings, complex implementation, and difficult reproducibility. Without standardization, comparisons can be unfair and insights inconclusive. To address this dilemma, we propose OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning that categorizes prevalent approaches into recurrent-based and recurrent-free models. OpenSTL provides a modular and extensible framework implementing various state-of-the-art methods. We conduct standard evaluations on datasets across various domains, including synthetic moving object trajectory, human motion, driving scenes, traffic flow and weather forecasting. Based on our observations, we provide a detailed analysis of how model architecture and dataset properties affect spatio-temporal predictive learning performance. Surprisingly, we find that recurrent-free models achieve a good balance between efficiency and performance than recurrent models. Thus, we further extend the common MetaFormers to boost recurrent-free spatial-temporal predictive learning. We open-source the code and models at https://github.com/chengtan9907/OpenSTL.