AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

TL;DR

This paper investigates how data filtering practices in pretraining large language models influence the social and geographic diversity of web-based training data, highlighting implicit biases and preferences.

Contribution

It introduces a new dataset of self-descriptions from website creators and analyzes the impact of quality and language filters on social and geographic representation.

Findings

Quality classifiers can act as topical filters.

Language identification may overlook English content from certain regions.

Filtering practices influence social and geographic diversity in training data.

Abstract

Large language models' (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation. However, decisions around what data is retained or removed during this initial stage are under-scrutinized. In our work, we ground web text, which is a popular pretraining data source, to its social and geographic contexts. We create a new dataset of 10.3 million self-descriptions of website creators, and extract information about who they are and where they are from: their topical interests, social roles, and geographic affiliations. Then, we conduct the first study investigating how ten "quality" and English language identification (langID) filters affect webpages that vary along these social dimensions. Our experiments illuminate a range of implicit preferences in data curation: we show that some quality classifiers act like topical domain filters, and langID can overlook English content from some regions of the world. Overall, we hope that our work will encourage a new line of research on pretraining data curation practices and its social implications.