PatchFormer: A Patch-Based Time Series Foundation Model with Hierarchical Masked Reconstruction and Cross-Domain Transfer Learning for Zero-Shot Multi-Horizon Forecasting

TL;DR

PatchFormer is a novel patch-based foundation model for time series forecasting that leverages hierarchical masked reconstruction and cross-domain transfer learning to achieve state-of-the-art zero-shot multi-horizon forecasting with minimal task-specific data.

Contribution

It introduces a patch-based architecture with hierarchical masked reconstruction and efficient transfer learning for time series forecasting, enabling zero-shot predictions across diverse domains.

Findings

Achieves 27.3% lower mean squared error on benchmarks.

Requires 94% less task-specific training data.

Processes sequences 3.8 times faster than full transformers.

Abstract

Time series forecasting is a fundamental problem with applications in climate, energy, healthcare, and finance. Many existing approaches require domain-specific feature engineering and substantial labeled data for each task. We introduce PatchFormer, a patch-based time series foundation model that uses hierarchical masked reconstruction for self-supervised pretraining and lightweight adapters for efficient transfer. PatchFormer segments time series into patches and learns multiscale temporal representations with learnable aggregation across temporal scales. Pretraining uses masked patch reconstruction with dynamic masking and objectives that encourage both local accuracy and global consistency, followed by cross-domain knowledge distillation. Experiments on 24 benchmark datasets spanning weather, energy, traffic, finance, and healthcare demonstrate state-of-the-art zero-shot multi-horizon forecasting, reducing mean squared error by 27.3 percent relative to strong baselines while requiring 94 percent less task-specific training data. The model exhibits near log-linear scaling with more pretraining data up to 100 billion points and processes length-512 sequences 3.8x faster than full-sequence transformers.