Zyda: A 1.3T Dataset for Open Language Modeling

TL;DR

Zyda is a large, high-quality open-source dataset of 1.3 trillion tokens designed to improve large language model training, outperforming existing datasets and enhancing model performance.

Contribution

Introduction of Zyda, a 1.3 trillion token open dataset with rigorous filtering, surpassing existing datasets and boosting LLM training effectiveness.

Findings

Zyda competes favorably with other open datasets.

Using Zyda improves model performance significantly.

Rigorous data filtering enhances dataset quality.

Abstract

The size of large language models (LLMs) has scaled dramatically in recent years and their computational and data requirements have surged correspondingly. State-of-the-art language models, even at relatively smaller sizes, typically require training on at least a trillion tokens. This rapid advancement has eclipsed the growth of open-source datasets available for large-scale LLM pretraining. In this paper, we introduce Zyda (Zyphra Dataset), a dataset under a permissive license comprising 1.3 trillion tokens, assembled by integrating several major respected open-source datasets into a single, high-quality corpus. We apply rigorous filtering and deduplication processes, both within and across datasets, to maintain and enhance the quality derived from the original datasets. Our evaluations show that Zyda not only competes favorably with other open datasets like Dolma, FineWeb, and RefinedWeb, but also substantially improves the performance of comparable models from the Pythia suite. Our rigorous data processing methods significantly enhance Zyda's effectiveness, outperforming even the best of its constituent datasets when used independently.