XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation

TL;DR

XicorAttention introduces a nonlinear correlation-based attention mechanism for time series forecasting, enhancing accuracy by capturing complex dependencies that traditional methods may miss.

Contribution

The paper proposes a novel attention mechanism using Chatterjee's rank correlation, integrated with SoftSort and SoftRank, to better model nonlinear dependencies in time series data.

Findings

Improved forecasting accuracy by up to 9.1% over existing models.

Effective integration of nonlinear correlation into Transformer architectures.

Demonstrated benefits on real-world datasets.

Abstract

Various Transformer-based models have been proposed for time series forecasting. These models leverage the self-attention mechanism to capture long-term temporal or variate dependencies in sequences. Existing methods can be divided into two approaches: (1) reducing computational cost of attention by making the calculations sparse, and (2) reshaping the input data to aggregate temporal features. However, existing attention mechanisms may not adequately capture inherent nonlinear dependencies present in time series data, leaving room for improvement. In this study, we propose a novel attention mechanism based on Chatterjee's rank correlation coefficient, which measures nonlinear dependencies between variables. Specifically, we replace the matrix multiplication in standard attention mechanisms with this rank coefficient to measure the query-key relationship. Since computing Chatterjee's correlation coefficient involves sorting and ranking operations, we introduce a differentiable approximation employing SoftSort and SoftRank. Our proposed mechanism, ``XicorAttention,'' integrates it into several state-of-the-art Transformer models. Experimental results on real-world datasets demonstrate that incorporating nonlinear correlation into the attention improves forecasting accuracy by up to approximately 9.1\% compared to existing models.