TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting

TL;DR

TempoPFN introduces a synthetic pre-trained linear RNN model for zero-shot time series forecasting, achieving competitive results efficiently without real data, and offers a reproducible pipeline for future research.

Contribution

The paper presents TempoPFN, a novel synthetic pre-training approach for linear RNNs that enables efficient zero-shot long-horizon time series forecasting.

Findings

Outperforms existing synthetic-only models on benchmarks.

Surpasses most real-data trained models in zero-shot settings.

Offers a fully parallelizable training and inference process.

Abstract

Foundation models for zero-shot time series forecasting face challenges in efficient long-horizon prediction and reproducibility, with existing synthetic-only approaches underperforming on challenging benchmarks. This paper presents TempoPFN, a univariate time series foundation model based on linear Recurrent Neural Networks (RNNs) pre-trained exclusively on synthetic data. The model uses a GatedDeltaProduct architecture with state-weaving for fully parallelizable training across sequence lengths, eliminating the need for windowing or summarization techniques while maintaining robust temporal state-tracking. Our comprehensive synthetic data pipeline unifies diverse generators, including stochastic differential equations, Gaussian processes, and audio synthesis, with novel augmentations. In zero-shot evaluations on the Gift-Eval, fev-bench and Chronos-ZS benchmarks, TempoPFN achieves top-tier competitive performance, outperforming all existing synthetic-only approaches and surpassing the majority of models trained on real-world data, while being more efficient than existing baselines by leveraging fully parallelizable training and inference. We open-source our complete data generation pipeline and training code, providing a reproducible foundation for future research.