EvoLM: In Search of Lost Language Model Training Dynamics

TL;DR

EvoLM provides a comprehensive suite for analyzing language model training dynamics across multiple stages, revealing key insights and trade-offs, and promoting transparency through open release of models and data.

Contribution

It introduces EvoLM, a systematic framework for analyzing LM training stages, and offers extensive empirical data and tools for understanding training impacts.

Findings

Diminishing returns from excessive pre-training and post-training.

Effective strategies for mitigating forgetting during domain-specific continued pre-training.

Continued pre-training is crucial for bridging pre-training and post-training phases.

Abstract

Modern language model (LM) training has been divided into multiple stages, making it difficult for downstream developers to evaluate the impact of design choices made at each stage. We present EvoLM, a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning. We train over 100 LMs with 1B and 4B parameters from scratch, and evaluate both upstream (language modeling) and downstream (problem-solving) capabilities, including considerations of both in-domain and out-of-domain generalization. Key insights highlight the diminishing returns from excessive pre-training and post-training, the importance and practices of mitigating forgetting during domain-specific continued pre-training, the crucial role of continued pre-training in bridging pre-training and post-training phases, and various intricate trade-offs when configuring supervised fine-tuning and reinforcement learning. To facilitate open research and reproducibility, we release all pre-trained and post-trained models, training datasets for all stages, and our entire training and evaluation pipeline.