Easy attention: A simple attention mechanism for temporal predictions with transformers

TL;DR

This paper introduces easy attention, a simplified attention mechanism for transformers that directly learns attention scores, improving robustness and performance in temporal predictions of chaotic systems compared to traditional self-attention and LSTM.

Contribution

The paper proposes a novel easy attention mechanism that eliminates the need for queries, keys, and softmax, directly learning attention scores for better temporal predictions.

Findings

Easy attention outperforms self attention and LSTM in chaotic system predictions.

The method demonstrates robustness across Lorenz system, turbulence shear flow, and nuclear reactor models.

Simplifies attention mechanism while maintaining or improving accuracy.

Abstract

To improve the robustness of transformer neural networks used for temporal-dynamics prediction of chaotic systems, we propose a novel attention mechanism called easy attention which we demonstrate in time-series reconstruction and prediction. While the standard self attention only makes use of the inner product of queries and keys, it is demonstrated that the keys, queries and softmax are not necessary for obtaining the attention score required to capture long-term dependencies in temporal sequences. Through the singular-value decomposition (SVD) on the softmax attention score, we further observe that self attention compresses the contributions from both queries and keys in the space spanned by the attention score. Therefore, our proposed easy-attention method directly treats the attention scores as learnable parameters. This approach produces excellent results when reconstructing and predicting the temporal dynamics of chaotic systems exhibiting more robustness and less complexity than self attention or the widely-used long short-term memory (LSTM) network. We show the improved performance of the easy-attention method in the Lorenz system, a turbulence shear flow and a model of a nuclear reactor.