Statistical benchmarking of transformer models in low signal-to-noise time-series forecasting

TL;DR

This paper evaluates transformer models for multivariate time-series forecasting in low-data, noisy environments, demonstrating their superiority over traditional methods and introducing a dynamic sparsification technique for improved performance.

Contribution

It introduces a two-way attention transformer architecture and a dynamic sparsification method, enhancing forecasting accuracy in low signal-to-noise scenarios with interpretable attention patterns.

Findings

Two-way attention transformers outperform baselines in noisy settings.

Dynamic sparsification improves model robustness in low SNR conditions.

Attention patterns reveal interpretable structures and regularization effects.

Abstract

We study the performance of transformer architectures for multivariate time-series forecasting in low-data regimes consisting of only a few years of daily observations. Using synthetically generated processes with known temporal and cross-sectional dependency structures and varying signal-to-noise ratios, we conduct bootstrapped experiments that enable direct evaluation via out-of-sample correlations with the optimal ground-truth predictor. We show that two-way attention transformers, which alternate between temporal and cross-sectional self-attention, can outperform standard baselines-Lasso, boosting methods, and fully connected multilayer perceptrons-across a wide range of settings, including low signal-to-noise regimes. We further introduce a dynamic sparsification procedure for attention matrices applied during training, and demonstrate that it becomes significantly effective in noisy environments, where the correlation between the target variable and the optimal predictor is on the order of a few percent. Analysis of the learned attention patterns reveals interpretable structure and suggests connections to sparsity-inducing regularization in classical regression, providing insight into why these models generalize effectively under noise.