Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting

TL;DR

This paper investigates how pre-trained language models can be effectively transferred to time series forecasting, highlighting the importance of design choices and the persistent transfer gap despite training.

Contribution

It provides a detailed analysis of transfer from language models to time series forecasting, revealing the ongoing benefits of pre-training over random initialization.

Findings

Transfer from language models improves time series forecasting performance.

Design choices significantly affect validation loss in low-data regimes.

A non-vanishing transfer gap persists even after models converge.

Abstract

Recent works have demonstrated the effectiveness of adapting pre-trained language models (LMs) for forecasting time series in the low-data regime. We build upon these findings by analyzing the effective transfer from language models to time series forecasting under various design choices including upstream post-training, time series tokenizer and language backbone size. In the low-data regime, these design choices have a significant impact on the validation loss, with clear-cut choices that outperform others. Contrary to Hernandez et al. (2021), we observe that the validation loss of the LMs continues to smoothly decrease long after the validation loss of the randomly initialized models has converged, leading to a non-vanishing transfer gap that holds across design choices. These findings not only help shed light on the effective use of compute-efficient training for time series, but also open the way for the study of modality-agnostic properties of data distributions leveraged by these models.