Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

TL;DR

This paper introduces a convolutional self-attention mechanism and a LogSparse Transformer model that enhance local context understanding and reduce memory usage, significantly improving long-term time series forecasting accuracy.

Contribution

It proposes convolutional self-attention to incorporate local context and LogSparse Transformer to address memory bottlenecks in long sequence modeling.

Findings

Outperforms state-of-the-art methods on real-world datasets

Reduces memory complexity to O(L(log L)^2)

Improves forecasting accuracy for long sequences

Abstract

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.