EnTransformer: A Deep Generative Transformer for Multivariate Probabilistic Forecasting

TL;DR

EnTransformer is a novel deep generative Transformer-based framework that models complex multivariate probabilistic forecasts directly, outperforming existing methods in calibration and accuracy across multiple benchmark datasets.

Contribution

It introduces a stochastic learning paradigm with energy-based scoring into Transformers for multivariate probabilistic forecasting, avoiding restrictive parametric assumptions.

Findings

Produces well-calibrated probabilistic forecasts.

Outperforms benchmark models on multiple datasets.

Effectively models long-range dependencies and cross-series interactions.

Abstract

Reliable uncertainty quantification is critical in multivariate time series forecasting problems arising in domains such as energy systems and transportation networks, among many others. Although Transformer-based architectures have recently achieved strong performance for sequence modeling, most probabilistic forecasting approaches rely on restrictive parametric likelihoods or quantile-based objectives. They can struggle to capture complex joint predictive distributions across multiple correlated time series. This work proposes EnTransformer, a deep generative forecasting framework that integrates engression, a stochastic learning paradigm for modeling conditional distributions, with the expressive sequence modeling capabilities of Transformers. The proposed approach injects stochastic noise into the model representation and optimizes an energy-based scoring objective to directly learn the conditional predictive distribution without imposing parametric assumptions. This design enables EnTransformer to generate coherent multivariate forecast trajectories while preserving Transformers' capacity to effectively model long-range temporal dependencies and cross-series interactions. We evaluate our proposed EnTransformer on several widely used benchmarks for multivariate probabilistic forecasting, including Electricity, Traffic, Solar, Taxi, KDD-cup, and Wikipedia datasets. Experimental results demonstrate that EnTransformer produces well-calibrated probabilistic forecasts and consistently outperforms the benchmark models.