ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer

TL;DR

ElasTST introduces a versatile, non-autoregressive time-series transformer that effectively handles varied forecasting horizons through innovative design features like structured self-attention, tunable position embeddings, and multi-scale patches.

Contribution

This paper presents ElasTST, a novel elastic transformer architecture for time-series forecasting that maintains invariance across different horizons during inference.

Findings

ElasTST outperforms state-of-the-art models in varied-horizon forecasting tasks.

The model demonstrates robustness and adaptability across multiple datasets.

Experimental results validate the effectiveness of its design components.

Abstract

Numerous industrial sectors necessitate models capable of providing robust forecasts across various horizons. Despite the recent strides in crafting specific architectures for time-series forecasting and developing pre-trained universal models, a comprehensive examination of their capability in accommodating varied-horizon forecasting during inference is still lacking. This paper bridges this gap through the design and evaluation of the Elastic Time-Series Transformer (ElasTST). The ElasTST model incorporates a non-autoregressive design with placeholders and structured self-attention masks, warranting future outputs that are invariant to adjustments in inference horizons. A tunable version of rotary position embedding is also integrated into ElasTST to capture time-series-specific periods and enhance adaptability to different horizons. Additionally, ElasTST employs a multi-scale patch design, effectively integrating both fine-grained and coarse-grained information. During the training phase, ElasTST uses a horizon reweighting strategy that approximates the effect of random sampling across multiple horizons with a single fixed horizon setting. Through comprehensive experiments and comparisons with state-of-the-art time-series architectures and contemporary foundation models, we demonstrate the efficacy of ElasTST's unique design elements. Our findings position ElasTST as a robust solution for the practical necessity of varied-horizon forecasting.