Frozen in Time: Parameter-Efficient Time Series Transformers via Reservoir-Induced Feature Expansion and Fixed Random Dynamics

TL;DR

FreezeTST introduces a hybrid Transformer model with frozen reservoir blocks and trainable layers, significantly reducing training costs while maintaining or improving long-term forecasting accuracy across multiple benchmarks.

Contribution

The paper presents a novel hybrid Transformer architecture that integrates frozen reservoir features with trainable layers, enhancing efficiency and performance in long-term time series forecasting.

Findings

Matches or surpasses specialized models like Informer and Autoformer.

Reduces training time and parameters without sacrificing accuracy.

Demonstrates effectiveness across seven long-term forecasting benchmarks.

Abstract

Transformers are the de-facto choice for sequence modelling, yet their quadratic self-attention and weak temporal bias can make long-range forecasting both expensive and brittle. We introduce FreezeTST, a lightweight hybrid that interleaves frozen random-feature (reservoir) blocks with standard trainable Transformer layers. The frozen blocks endow the network with rich nonlinear memory at no optimisation cost; the trainable layers learn to query this memory through self-attention. The design cuts trainable parameters and also lowers wall-clock training time, while leaving inference complexity unchanged. On seven standard long-term forecasting benchmarks, FreezeTST consistently matches or surpasses specialised variants such as Informer, Autoformer, and PatchTST; with substantially lower compute. Our results show that embedding reservoir principles within Transformers offers a simple, principled route to efficient long-term time-series prediction.