FreIE: Low-Frequency Spectral Bias in Neural Networks for Time-Series Tasks

TL;DR

This paper investigates the universal spectral bias in neural networks for time-series prediction, demonstrating its prevalence across models and proposing a regularization method, FreLE, to improve long-term forecasting performance.

Contribution

The paper uncovers the universal presence of spectral bias in neural networks for time-series tasks and introduces FreLE, a plug-and-play frequency regularization method to enhance model generalization.

Findings

Spectral bias is present in nearly all tested models.

FreLE improves long-term prediction accuracy.

FreLE outperforms baseline models in experiments.

Abstract

The inherent autocorrelation of time series data presents an ongoing challenge to multivariate time series prediction. Recently, a widely adopted approach has been the incorporation of frequency domain information to assist in long-term prediction tasks. Many researchers have independently observed the spectral bias phenomenon in neural networks, where models tend to fit low-frequency signals before high-frequency ones. However, these observations have often been attributed to the specific architectures designed by the researchers, rather than recognizing the phenomenon as a universal characteristic across models. To unify the understanding of the spectral bias phenomenon in long-term time series prediction, we conducted extensive empirical experiments to measure spectral bias in existing mainstream models. Our findings reveal that virtually all models exhibit this phenomenon. To mitigate the impact of spectral bias, we propose the FreLE (Frequency Loss Enhancement) algorithm, which enhances model generalization through both explicit and implicit frequency regularization. This is a plug-and-play model loss function unit. A large number of experiments have proven the superior performance of FreLE. Code is available at https://github.com/Chenxing-Xuan/FreLE.