CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting

TL;DR

CAPS introduces a novel structured attention mechanism for time series forecasting that effectively separates global trends, local shocks, and seasonal patterns, outperforming existing methods with linear complexity.

Contribution

The paper proposes CAPS, a new attention mechanism combining phase alignment and multiple gating paths, with a learned temporal weighting, advancing time series forecasting models.

Findings

Outperforms vanilla softmax and linear attention on benchmarks.

Achieves competitive results against seven strong baselines.

Maintains linear computational complexity.

Abstract

This paper presents $\textbf{CAPS}$ (Clock-weighted Aggregation with Prefix-products and Softmax), a structured attention mechanism for time series forecasting that decouples three distinct temporal structures: global trends, local shocks, and seasonal patterns. Standard softmax attention entangles these through global normalization, while recent recurrent models sacrifice long-term, order-independent selection for order-dependent causal structure. CAPS combines SO(2) rotations for phase alignment with three additive gating paths -- Riemann softmax, prefix-product gates, and a Clock baseline -- within a single attention layer. We introduce the Clock mechanism, a learned temporal weighting that modulates these paths through a shared notion of temporal importance. Experiments on long- and short-term forecasting benchmarks surpass vanilla softmax and linear attention mechanisms and demonstrate competitive performance against seven strong baselines with linear complexity. Our code implementation is available at https://github.com/vireshpati/CAPS-Attention.