Sensorformer: Cross-patch attention with global-patch compression is effective for high-dimensional multivariate time series forecasting

TL;DR

Sensorformer introduces a novel Transformer architecture that effectively fuses cross-time and cross-variable dependencies in high-dimensional multivariate time series forecasting by global patch compression and cross-patch attention, outperforming existing methods.

Contribution

It proposes Sensorformer, a new Transformer model that captures inter-variable correlations and causal relationships despite dynamic lags, with reduced computational complexity.

Findings

Sensorformer outperforms existing models on 9 real-world datasets.

It effectively captures dynamic causal relationships between variables.

The model reduces computational complexity compared to pure cross-patch self-attention.

Abstract

Among the existing Transformer-based multivariate time series forecasting methods, iTransformer, which treats each variable sequence as a token and only explicitly extracts cross-variable dependencies, and PatchTST, which adopts a channel-independent strategy and only explicitly extracts cross-time dependencies, both significantly outperform most Channel-Dependent Transformer that simultaneously extract cross-time and cross-variable dependencies. This indicates that existing Transformer-based multivariate time series forecasting methods still struggle to effectively fuse these two types of information. We attribute this issue to the dynamic time lags in the causal relationships between different variables. Therefore, we propose a new multivariate time series forecasting Transformer, Sensorformer, which first compresses the global patch information and then simultaneously extracts cross-variable and cross-time dependencies from the compressed representations. Sensorformer can effectively capture the correct inter-variable correlations and causal relationships, even in the presence of dynamic causal lags between variables, while also reducing the computational complexity of pure cross-patch self-attention from $O(D^2 \cdot Patch\_num^2 \cdot d\_model)$ to $O(D^2 \cdot Patch\_num \cdot d\_model)$. Extensive comparative and ablation experiments on 9 mainstream real-world multivariate time series forecasting datasets demonstrate the superiority of Sensorformer. The implementation of Sensorformer, following the style of the Time-series-library and scripts for reproducing the main results, is publicly available at https://github.com/BigYellowTiger/Sensorformer