2 OLMo 2 Furious

TL;DR

OLMo 2 introduces a family of fully open, scalable language models with improved training stability, specialized data mixtures, and open-instruct variants that outperform or match leading models while maintaining transparency and efficiency.

Contribution

This work presents OLMo 2's architecture, training techniques, specialized data mix, and open-instruct models, advancing open language model development with enhanced performance and transparency.

Findings

OLMo 2 models outperform or match leading open-weight models.

Specialized Dolmino Mix 1124 improves downstream task performance.

Open-instruct OLMo 2 models are competitive with proprietary models.

Abstract

We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes a family of dense autoregressive language models at 7B, 13B and 32B scales with fully released artifacts -- model weights, full training data, training code and recipes, training logs and thousands of intermediate checkpoints. In this work, we describe our modified model architecture and training recipe, focusing on techniques for achieving better training stability and improved per-token efficiency. Our updated pretraining data mixture introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best practices from T\"ulu 3 to develop OLMo 2-Instruct, focusing on permissive data and extending our final-stage reinforcement learning with verifiable rewards (RLVR). Our OLMo 2 base models sit at the Pareto frontier of performance to training compute, often matching or outperforming open-weight only models like Llama 3.1, Qwen 2.5, and Gemma 2 while using fewer FLOPs and with fully transparent training data, code, and recipe. Our fully open OLMo 2-Instruct models are competitive with open-weight only models of comparable size and even some proprietary models like GPT-3.5 Turbo and GPT 4o Mini.