Triplet Attention Transformer for Spatiotemporal Predictive Learning

TL;DR

This paper introduces a triplet attention transformer with a novel attention module that captures complex spatiotemporal dependencies, outperforming existing methods in various predictive learning scenarios.

Contribution

The paper proposes the Triplet Attention Module (TAM) within a transformer framework to effectively model inter-frame and intra-frame features for spatiotemporal prediction.

Findings

Achieves state-of-the-art performance across multiple scenarios.

Outperforms recurrent and other transformer-based methods.

Effectively captures long-range dependencies in space and time.

Abstract

Spatiotemporal predictive learning offers a self-supervised learning paradigm that enables models to learn both spatial and temporal patterns by predicting future sequences based on historical sequences. Mainstream methods are dominated by recurrent units, yet they are limited by their lack of parallelization and often underperform in real-world scenarios. To improve prediction quality while maintaining computational efficiency, we propose an innovative triplet attention transformer designed to capture both inter-frame dynamics and intra-frame static features. Specifically, the model incorporates the Triplet Attention Module (TAM), which replaces traditional recurrent units by exploring self-attention mechanisms in temporal, spatial, and channel dimensions. In this configuration: (i) temporal tokens contain abstract representations of inter-frame, facilitating the capture of inherent temporal dependencies; (ii) spatial and channel attention combine to refine the intra-frame representation by performing fine-grained interactions across spatial and channel dimensions. Alternating temporal, spatial, and channel-level attention allows our approach to learn more complex short- and long-range spatiotemporal dependencies. Extensive experiments demonstrate performance surpassing existing recurrent-based and recurrent-free methods, achieving state-of-the-art under multi-scenario examination including moving object trajectory prediction, traffic flow prediction, driving scene prediction, and human motion capture.