Grouped self-attention mechanism for a memory-efficient Transformer

TL;DR

This paper introduces a memory-efficient Transformer model with novel Grouped Self-Attention and Compressed Cross-Attention modules, effectively capturing long-range dependencies in time-series data while reducing computational complexity.

Contribution

The paper presents two new modules that enable Transformers to handle long sequences efficiently with linear complexity, improving time-series forecasting performance.

Findings

Achieved $O(l)$ complexity with sequence length $l$

Model performs comparably or better than existing methods

Effectively captures local and global information in time-series data

Abstract

Time-series data analysis is important because numerous real-world tasks such as forecasting weather, electricity consumption, and stock market involve predicting data that vary over time. Time-series data are generally recorded over a long period of observation with long sequences owing to their periodic characteristics and long-range dependencies over time. Thus, capturing long-range dependency is an important factor in time-series data forecasting. To solve these problems, we proposed two novel modules, Grouped Self-Attention (GSA) and Compressed Cross-Attention (CCA). With both modules, we achieved a computational space and time complexity of order $O(l)$ with a sequence length $l$ under small hyperparameter limitations, and can capture locality while considering global information. The results of experiments conducted on time-series datasets show that our proposed model efficiently exhibited reduced computational complexity and performance comparable to or better than existing methods.