Scaling LLM Pre-training with Vocabulary Curriculum

TL;DR

This paper introduces vocabulary curriculum learning for language models, dynamically expanding vocabularies during pretraining to improve efficiency and transferability, with promising results on small GPT models.

Contribution

It proposes a novel method for adaptive vocabulary expansion during pretraining, bridging the gap between static vocabularies and human-like language learning.

Findings

Log-linear scaling gains with vocabulary size

Improved pretraining efficiency on small GPT models

Effective transferability across tokenization granularities

Abstract

Modern language models rely on static vocabularies, fixed before pretraining, in contrast to the adaptive vocabulary acquisition observed in human language learning. To bridge this gap, we introduce vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size. Our method alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities. This approach naturally gives rise to an optimal computation allocation pattern: longer tokens capture predictable content, while shorter tokens focus on more complex, harder-to-predict contexts. Experiments on small-scale GPT models demonstrate improved scaling efficiency, reinforcing the effectiveness of dynamic tokenization. We release our code to support further research and plan to extend our experiments to larger models and diverse domains.