EMAformer: Enhancing Transformer through Embedding Armor for Time Series Forecasting

TL;DR

EMAformer enhances Transformer models for multivariate time series forecasting by introducing an auxiliary embedding suite with inductive biases, achieving state-of-the-art results and reducing forecasting errors across multiple benchmarks.

Contribution

The paper introduces EMAformer, a novel Transformer-based model with an auxiliary embedding suite and three inductive biases, improving stability and performance in time series forecasting.

Findings

Achieves state-of-the-art performance on 12 benchmarks.

Reduces forecasting errors by 2.73% in MSE and 5.15% in MAE.

Outperforms existing Transformer and MLP-based models.

Abstract

Multivariate time series forecasting is crucial across a wide range of domains. While presenting notable progress for the Transformer architecture, iTransformer still lags behind the latest MLP-based models. We attribute this performance gap to unstable inter-channel relationships. To bridge this gap, we propose EMAformer, a simple yet effective model that enhances the Transformer with an auxiliary embedding suite, akin to armor that reinforces its ability. By introducing three key inductive biases, i.e., \textit{global stability}, \textit{phase sensitivity}, and \textit{cross-axis specificity}, EMAformer unlocks the further potential of the Transformer architecture, achieving state-of-the-art performance on 12 real-world benchmarks and reducing forecasting errors by an average of 2.73\% in MSE and 5.15\% in MAE. This significantly advances the practical applicability of Transformer-based approaches for multivariate time series forecasting. The code is available on https://github.com/PlanckChang/EMAformer.