Baguan-TS: A Sequence-Native In-Context Learning Model for Time Series Forecasting with Covariates

TL;DR

Baguan-TS introduces a sequence-native in-context learning model for time series forecasting that effectively integrates raw sequence representations with transformers, achieving superior performance and robustness on benchmark datasets.

Contribution

This work presents a unified framework combining sequence-native learning with in-context learning in transformers, addressing calibration and oversmoothing issues for improved time series forecasting.

Findings

Outperforms established baselines on public benchmarks with covariates

Achieves significant improvements in point and probabilistic forecasting metrics

Demonstrates robustness across diverse real-world energy datasets

Abstract

Transformers enable in-context learning (ICL) for rapid, gradient-free adaptation in time series forecasting, yet most ICL-style approaches rely on tabularized, hand-crafted features, while end-to-end sequence models lack inference-time adaptation. We bridge this gap with a unified framework, Baguan-TS, which integrates the raw-sequence representation learning with ICL, instantiated by a 3D Transformer that attends jointly over temporal, variable, and context axes. To make this high-capacity model practical, we tackle two key hurdles: (i) calibration and training stability, improved with a feature-agnostic, target-space retrieval-based local calibration; and (ii) output oversmoothing, mitigated via context-overfitting strategy. On public benchmark with covariates, Baguan-TS consistently outperforms established baselines, achieving the highest win rate and significant reductions in both point and probabilistic forecasting metrics. Further evaluations across diverse real-world energy datasets demonstrate its robustness, yielding substantial improvements.