Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers

TL;DR

This paper presents a pure transformer model for autoregressive video prediction of physical simulations, extending prediction horizons and enabling interpretability without complex training or latent features.

Contribution

It introduces a simple, parameter-efficient transformer approach for pixel-space video prediction that outperforms latent-space methods in physical simulation tasks.

Findings

Extends prediction horizon by up to 50% compared to latent-space models

Maintains comparable video quality metrics

Enables interpretability and generalization to out-of-distribution parameters

Abstract

Inspired by the performance and scalability of autoregressive large language models (LLMs), transformer-based models have seen recent success in the visual domain. This study investigates a transformer adaptation for video prediction with a simple end-to-end approach, comparing various spatiotemporal self-attention layouts. Focusing on causal modeling of physical simulations over time; a common shortcoming of existing video-generative approaches, we attempt to isolate spatiotemporal reasoning via physical object tracking metrics and unsupervised training on physical simulation datasets. We introduce a simple yet effective pure transformer model for autoregressive video prediction, utilizing continuous pixel-space representations for video prediction. Without the need for complex training strategies or latent feature-learning components, our approach significantly extends the time horizon for physically accurate predictions by up to 50% when compared with existing latent-space approaches, while maintaining comparable performance on common video quality metrics. In addition, we conduct interpretability experiments to identify network regions that encode information useful to perform accurate estimations of PDE simulation parameters via probing models, and find that this generalizes to the estimation of out-of-distribution simulation parameters. This work serves as a platform for further attention-based spatiotemporal modeling of videos via a simple, parameter efficient, and interpretable approach.