Zyda-2: a 5 Trillion Token High-Quality Dataset

TL;DR

Zyda-2 is a massive five trillion token dataset designed for pretraining language models, constructed from high-quality open-source sources and optimized through filtering, enabling state-of-the-art performance in its class.

Contribution

This paper introduces Zyda-2, a large-scale high-quality dataset for language model pretraining, and demonstrates its effectiveness by training state-of-the-art models.

Findings

Zyda-2 enables training of state-of-the-art models.

The dataset is publicly available under an open license.

High-quality filtering improves model performance.

Abstract

In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2