Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models

TL;DR

This paper investigates how tokenizer design and pretraining influence time series forecasting models, showing that well-designed tokenizers combined with transfer learning significantly improve performance, especially with small vocabularies.

Contribution

It provides a systematic analysis of tokenizer scaling and quantization effects, demonstrating the importance of alignment between tokenization and pretraining in time series models.

Findings

Pretrained models benefit more from well-designed tokenizers.

Misaligned tokenization can negate pretraining advantages.

Small, efficient vocabularies with pretraining excel in multi-modal forecasting.

Abstract

Tokenization and transfer learning are two critical components in building state of the art time series foundation models for forecasting. In this work, we systematically study the effect of tokenizer design, specifically scaling and quantization strategies, on model performance, alongside the impact of pretraining versus random initialization. We show that tokenizer configuration primarily governs the representational capacity and stability of the model, while transfer learning influences optimization efficiency and alignment. Using a combination of empirical training experiments and theoretical analyses, we demonstrate that pretrained models consistently leverage well-designed tokenizers more effectively, particularly at smaller vocabulary sizes. Conversely, misaligned tokenization can diminish or even invert the benefits of pretraining. These findings highlight the importance of careful tokenization in time series modeling and suggest that combining small, efficient vocabularies with pretrained weights is especially advantageous in multi-modal forecasting settings, where the overall vocabulary must be shared across modalities. Our results provide concrete guidance for designing tokenizers and leveraging transfer learning in discrete representation learning for continuous signals.