FuLG: 150B Romanian Corpus for Language Model Pretraining

TL;DR

FuLG is a large-scale Romanian corpus of 150 billion tokens, created from CommonCrawl, with a detailed filtering methodology and comparative analysis against existing Romanian datasets.

Contribution

This paper introduces FuLG, a massive Romanian corpus for language model pretraining, along with a novel filtering process and ablation studies demonstrating its effectiveness.

Findings

FuLG contains 150 billion tokens.

Filtering methodology improves corpus quality.

Compared favorably against existing Romanian corpora.

Abstract

Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for filtering FuLG and compare it via ablation studies against existing Romanian corpora.