Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages

TL;DR

This study evaluates the quality of four major web-crawled corpora across eleven European languages, revealing that higher intrinsic quality does not necessarily translate into better language model performance on downstream tasks.

Contribution

It provides a comprehensive comparison of web-crawled corpora quality and their actual impact on language model training and performance across multiple languages.

Findings

MaCoCu and OSCAR have the highest intrinsic quality scores.

CC100 corpus yields the best downstream task performance.

Corpus quality differences have limited effect on language model outcomes.

Abstract

Large, curated, web-crawled corpora play a vital role in training language models (LMs). They form the lion's share of the training data in virtually all recent LMs, such as the well-known GPT, LLaMA and XLM-RoBERTa models. However, despite this importance, relatively little attention has been given to the quality of these corpora. In this paper, we compare four of the currently most relevant large, web-crawled corpora (CC100, MaCoCu, mC4 and OSCAR) across eleven lower-resourced European languages. Our approach is two-fold: first, we perform an intrinsic evaluation by performing a human evaluation of the quality of samples taken from different corpora; then, we assess the practical impact of the qualitative differences by training specific LMs on each of the corpora and evaluating their performance on downstream tasks. We find that there are clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining the best results. However, during the extrinsic evaluation, we actually find that the CC100 corpus achieves the highest scores. We conclude that, in our experiments, the quality of the web-crawled corpora does not seem to play a significant role when training LMs.