Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting

TL;DR

This paper demonstrates that linear attention in Transformers can be interpreted as VAR models for time series forecasting, and introduces SAMoVAR, a variant that improves interpretability and performance by aligning architecture with autoregressive objectives.

Contribution

It reveals the VAR structure within linear attention, identifies structural mismatches in multi-layer Transformers, and proposes SAMoVAR, a new model that aligns Transformer architecture with autoregressive forecasting.

Findings

Single linear attention layer models VAR structures.

Multi-layer Transformers can be aligned as VAR models.

SAMoVAR outperforms state-of-the-art TSF models in accuracy and interpretability.

Abstract

Autoregressive attention-based time series forecasting (TSF) has drawn increasing interest, with mechanisms like linear attention sometimes outperforming vanilla attention. However, deeper Transformer architectures frequently misalign with autoregressive objectives, obscuring the underlying VAR structure embedded within linear attention and hindering their ability to capture the data generative processes in TSF. In this work, we first show that a single linear attention layer can be interpreted as a dynamic vector autoregressive (VAR) structure. We then explain that existing multi-layer Transformers have structural mismatches with the autoregressive forecasting objective, which impair interpretability and generalization ability. To address this, we show that by rearranging the MLP, attention, and input-output flow, multi-layer linear attention can also be aligned as a VAR model. Then, we propose Structural Aligned Mixture of VAR (SAMoVAR), a linear Transformer variant that integrates interpretable dynamic VAR weights for multivariate TSF. By aligning the Transformer architecture with autoregressive objectives, SAMoVAR delivers improved performance, interpretability, and computational efficiency, comparing to SOTA TSF models.