Fredformer: Frequency Debiased Transformer for Time Series Forecasting

TL;DR

Fredformer is a Transformer-based model for time series forecasting that mitigates frequency bias by learning features across all frequency bands, improving accuracy especially on high-frequency data.

Contribution

The paper introduces Fredformer, a novel Transformer framework that addresses frequency bias in time series forecasting, with a lightweight variant for efficiency.

Findings

Outperforms baseline models on real-world datasets

Effectively captures high-frequency features

Lightweight variant maintains performance with fewer resources

Abstract

The Transformer model has shown leading performance in time series forecasting. Nevertheless, in some complex scenarios, it tends to learn low-frequency features in the data and overlook high-frequency features, showing a frequency bias. This bias prevents the model from accurately capturing important high-frequency data features. In this paper, we undertook empirical analyses to understand this bias and discovered that frequency bias results from the model disproportionately focusing on frequency features with higher energy. Based on our analysis, we formulate this bias and propose Fredformer, a Transformer-based framework designed to mitigate frequency bias by learning features equally across different frequency bands. This approach prevents the model from overlooking lower amplitude features important for accurate forecasting. Extensive experiments show the effectiveness of our proposed approach, which can outperform other baselines in different real-world time-series datasets. Furthermore, we introduce a lightweight variant of the Fredformer with an attention matrix approximation, which achieves comparable performance but with much fewer parameters and lower computation costs. The code is available at: https://github.com/chenzRG/Fredformer