PFformer: A Position-Free Transformer Variant for Extreme-Adaptive Multivariate Time Series Forecasting

TL;DR

PFformer is a novel position-free Transformer model that improves multivariate time series forecasting, especially for datasets with extreme variability and rare events, by using innovative embedding strategies to better capture inter-variable dependencies.

Contribution

The paper introduces PFformer, a position-free Transformer variant with two novel embeddings, enhancing dependency modeling and forecasting accuracy in challenging MTS datasets.

Findings

PFformer outperforms state-of-the-art models by 20-60% in forecasting accuracy.

It effectively captures complex inter-variable dependencies without positional encoding.

PFformer demonstrates robustness on datasets with extreme variability and rare events.

Abstract

Multivariate time series (MTS) forecasting is vital in fields like weather, energy, and finance. However, despite deep learning advancements, traditional Transformer-based models often diminish the effect of crucial inter-variable relationships by singular token embedding and struggle to effectively capture complex dependencies among variables, especially in datasets with rare or extreme events. These events create significant imbalances and lead to high skewness, complicating accurate prediction efforts. This study introduces PFformer, a position-free Transformer-based model designed for single-target MTS forecasting, specifically for challenging datasets characterized by extreme variability. PFformer integrates two novel embedding strategies: Enhanced Feature-based Embedding (EFE) and Auto-Encoder-based Embedding (AEE). EFE effectively encodes inter-variable dependencies by mapping related sequence subsets to high-dimensional spaces without positional constraints, enhancing the encoder's functionality. PFformer shows superior forecasting accuracy without the traditional limitations of positional encoding in MTS modeling. We evaluated PFformer across four challenging datasets, focusing on two key forecasting scenarios: long sequence prediction for 3 days ahead and rolling predictions every four hours to reflect real-time decision-making processes in water management. PFformer demonstrated remarkable improvements, from 20% to 60%, compared with state-of-the-art models.