Beyond Line-Level Filtering for the Pretraining Corpora of LLMs

TL;DR

This paper introduces pattern-aware filtering methods for pretraining data that improve language model performance by better retaining valuable content during data cleaning.

Contribution

It proposes two novel filtering techniques, PLD and PTF, which consider sequential distribution to preserve important content often lost in traditional filtering.

Findings

Improved performance on multiple-choice benchmarks.

Enhanced generative question-answering accuracy on SQuAD v1 and KorQuAD v1.

Consistent gains across English and Korean language models.

Abstract

While traditional line-level filtering techniques, such as line-level deduplication and trailing-punctuation filters, are commonly used, these basic methods can sometimes discard valuable content, negatively affecting downstream performance. In this paper, we introduce two methods-pattern-aware line-level deduplication (PLD) and pattern-aware trailing punctuation filtering (PTF)-by enhancing the conventional filtering techniques. Our approach not only considers line-level signals but also takes into account their sequential distribution across documents, enabling us to retain structurally important content that might otherwise be removed. We evaluate these proposed methods by training small language models (1 B parameters) in both English and Korean. The results demonstrate that our methods consistently improve performance on multiple-choice benchmarks and significantly enhance generative question-answering accuracy on both SQuAD v1 and KorQuAD v1.