2 OLMo 2 Furious
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri

TL;DR
OLMo 2 introduces a family of fully open, scalable language models with improved training stability, specialized data mixtures, and open-instruct variants that outperform or match leading models while maintaining transparency and efficiency.
Contribution
This work presents OLMo 2's architecture, training techniques, specialized data mix, and open-instruct models, advancing open language model development with enhanced performance and transparency.
Findings
OLMo 2 models outperform or match leading open-weight models.
Specialized Dolmino Mix 1124 improves downstream task performance.
Open-instruct OLMo 2 models are competitive with proprietary models.
Abstract
We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes a family of dense autoregressive language models at 7B, 13B and 32B scales with fully released artifacts -- model weights, full training data, training code and recipes, training logs and thousands of intermediate checkpoints. In this work, we describe our modified model architecture and training recipe, focusing on techniques for achieving better training stability and improved per-token efficiency. Our updated pretraining data mixture introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best practices from T\"ulu 3 to develop OLMo 2-Instruct, focusing on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
- 🤗allenai/OLMo-2-1124-7Bmodel· 63k dl· ♡ 6563k dl♡ 65
- 🤗allenai/OLMo-2-1124-13Bmodel· 7.5k dl· ♡ 687.5k dl♡ 68
- 🤗allenai/OLMo-2-1124-7B-SFT-Previewmodel· 21 dl· ♡ 321 dl♡ 3
- 🤗allenai/OLMo-2-1124-7B-DPO-Previewmodel· 28 dl· ♡ 228 dl♡ 2
- 🤗allenai/OLMo-2-1124-13B-SFT-Previewmodel· 16 dl· ♡ 316 dl♡ 3
- 🤗allenai/OLMo-2-1124-13B-DPO-Previewmodel· 22 dl· ♡ 322 dl♡ 3
- 🤗allenai/OLMo-2-1124-7B-Instruct-previewmodel· 35 dl· ♡ 4735 dl♡ 47
- 🤗allenai/OLMo-2-1124-13B-Instruct-previewmodel· 48 dl· ♡ 5848 dl♡ 58
- 🤗cortexso/olmo-2model· 184 dl· ♡ 1184 dl♡ 1
- 🤗allenai/OLMo-2-1124-7B-Instructmodel· 18k dl· ♡ 4818k dl♡ 48
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Artificial Intelligence in Healthcare and Education · Natural Language Processing Techniques
MethodsBalanced Selection · LLaMA
