DRAformer: Differentially Reconstructed Attention Transformer for Time-Series Forecasting

TL;DR

DRAformer is a novel transformer-based model for time-series forecasting that enhances feature stability and temporal dependency learning through differenced sequences and reconstructed attention mechanisms, leading to superior performance.

Contribution

The paper introduces DRAformer, which employs differenced sequences and innovative reconstructed attention to improve time-series forecasting accuracy over existing models.

Findings

DRAformer outperforms state-of-the-art baselines on four large-scale datasets.

The differenced sequence learning improves stability and feature clarity.

Reconstructed attention mechanisms effectively focus on significant sequence associations.

Abstract

Time-series forecasting plays an important role in many real-world scenarios, such as equipment life cycle forecasting, weather forecasting, and traffic flow forecasting. It can be observed from recent research that a variety of transformer-based models have shown remarkable results in time-series forecasting. However, there are still some issues that limit the ability of transformer-based models on time-series forecasting tasks: (i) learning directly on raw data is susceptible to noise due to its complex and unstable feature representation; (ii) the self-attention mechanisms pay insufficient attention to changing features and temporal dependencies. In order to solve these two problems, we propose a transformer-based differentially reconstructed attention model DRAformer. Specifically, DRAformer has the following innovations: (i) learning against differenced sequences, which preserves clear and stable sequence features by differencing and highlights the changing properties of sequences; (ii) the reconstructed attention: integrated distance attention exhibits sequential distance through a learnable Gaussian kernel, distributed difference attention calculates distribution difference by mapping the difference sequence to the adaptive feature space, and the combination of the two effectively focuses on the sequences with prominent associations; (iii) the reconstructed decoder input, which extracts sequence features by integrating variation information and temporal correlations, thereby obtaining a more comprehensive sequence representation. Extensive experiments on four large-scale datasets demonstrate that DRAformer outperforms state-of-the-art baselines.