dots.llm1 Technical Report

TL;DR

dots.llm1 is a large-scale Mixture of Experts language model with 14B active parameters, achieving competitive performance with reduced costs, and providing open-source checkpoints for research.

Contribution

We introduce dots.llm1, a 142B parameter MoE model with efficient training, high performance, and open-source checkpoints, advancing large language model research.

Findings

Achieves performance comparable to larger models like Qwen2.5-72B.

Reduces training and inference costs through MoE architecture.

Provides open-source checkpoints at every trillion tokens.

Abstract

Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this report, we present dots.llm1, a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on 11.2T high-quality tokens and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.