Linear Attention is Enough in Spatial-Temporal Forecasting

TL;DR

This paper introduces a novel Transformer-based approach for spatial-temporal traffic forecasting, using independent tokens for nodes over time and a Nyström-based variant for linear complexity, achieving state-of-the-art results.

Contribution

Proposes STformer and NSTformer models that effectively capture spatial-temporal patterns with improved efficiency and accuracy in traffic forecasting.

Findings

Achieves state-of-the-art performance on traffic datasets.

NSTformer offers linear complexity with competitive accuracy.

Models outperform existing methods in capturing dynamic road network topology.

Abstract

As the most representative scenario of spatial-temporal forecasting tasks, the traffic forecasting task attracted numerous attention from machine learning community due to its intricate correlation both in space and time dimension. Existing methods often treat road networks over time as spatial-temporal graphs, addressing spatial and temporal representations independently. However, these approaches struggle to capture the dynamic topology of road networks, encounter issues with message passing mechanisms and over-smoothing, and face challenges in learning spatial and temporal relationships separately. To address these limitations, we propose treating nodes in road networks at different time steps as independent spatial-temporal tokens and feeding them into a vanilla Transformer to learn complex spatial-temporal patterns, design \textbf{STformer} achieving SOTA. Given its quadratic complexity, we introduce a variant \textbf{NSTformer} based on Nystr$\ddot{o}$m method to approximate self-attention with linear complexity but even slightly better than former in a few cases astonishingly. Extensive experimental results on traffic datasets demonstrate that the proposed method achieves state-of-the-art performance at an affordable computational cost. Our code is available at \href{https://github.com/XinyuNing/STformer-and-NSTformer}{https://github.com/XinyuNing/STformer-and-NSTformer}.