Datasheet for the Pile

TL;DR

This paper introduces 'The Pile', a large 825 GiB dataset of diverse human-authored text sources designed for training large-scale language models, providing detailed documentation for transparency and reproducibility.

Contribution

It presents a comprehensive dataset with detailed datasheet documentation, enabling better understanding and use in large-scale language model training.

Findings

The dataset contains 22 diverse text sources.

The dataset size is 825 GiB.

Provides transparency through detailed datasheet documentation.

Abstract

This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online.