Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Equality

TL;DR

This paper introduces a linear attention mechanism for multivariate time series that reduces computational complexity by leveraging entropy equality, enabling scalable and efficient modeling with competitive forecasting results.

Contribution

The work presents a novel entropy-based linear attention mechanism that significantly improves scalability for long sequences in time series modeling.

Findings

Achieves linear complexity in attention computation.

Demonstrates superior or comparable forecasting accuracy.

Reduces memory usage and computational time substantially.

Abstract

Attention mechanisms have been extensively employed in various applications, including time series modeling, owing to their capacity to capture intricate dependencies; however, their utility is often constrained by quadratic computational complexity, which impedes scalability for long sequences. In this work, we propose a novel linear attention mechanism designed to overcome these limitations. Our approach is grounded in a theoretical demonstration that entropy, as a strictly concave function on the probability simplex, implies that distributions with aligned probability rankings and similar entropy values exhibit structural resemblance. Building on this insight, we develop an efficient approximation algorithm that computes the entropy of dot-product-derived distributions with only linear complexity, enabling the implementation of a linear attention mechanism based on entropy equality. Through rigorous analysis, we reveal that the effectiveness of attention in spatio-temporal time series modeling may not primarily stem from the non-linearity of softmax but rather from the attainment of a moderate and well-balanced weight distribution. Extensive experiments on four spatio-temporal datasets validate our method, demonstrating competitive or superior forecasting performance while achieving substantial reductions in both memory usage and computational time.