Enhancing Spatiotemporal Prediction Model using Modular Design and Beyond

TL;DR

This paper introduces a modular approach to spatiotemporal prediction that separates spatial encoding from future prediction, leading to improved performance and state-of-the-art results on benchmark datasets.

Contribution

It proposes a novel modular design decomposing the model into spatial encoder-decoder and predictor modules, reducing complexity and enhancing convergence.

Findings

Achieved state-of-the-art results on KTH-Action and MovingMNIST datasets.

Improved computational efficiency compared to traditional joint models.

Demonstrated robustness and accuracy of the modular approach.

Abstract

Predictive learning uses a known state to generate a future state over a period of time. It is a challenging task to predict spatiotemporal sequence because the spatiotemporal sequence varies both in time and space. The mainstream method is to model spatial and temporal structures at the same time using RNN-based or transformer-based architecture, and then generates future data by using learned experience in the way of auto-regressive. The method of learning spatial and temporal features simultaneously brings a lot of parameters to the model, which makes the model difficult to be convergent. In this paper, a modular design is proposed, which decomposes spatiotemporal sequence model into two modules: a spatial encoder-decoder and a predictor. These two modules can extract spatial features and predict future data respectively. The spatial encoder-decoder maps the data into a latent embedding space and generates data from the latent space while the predictor forecasts future embedding from past. By applying the design to the current research and performing experiments on KTH-Action and MovingMNIST datasets, we both improve computational performance and obtain state-of-the-art results.