FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering

TL;DR

This paper presents an LLM-based line-level filtering method that improves data quality for training large language models, leading to better performance and efficiency, and releases a new annotated dataset for the community.

Contribution

Introduces a novel LLM-based line filtering approach and a labeled dataset to enhance training data quality for LLMs.

Findings

Filtered data improves GPT-2 model accuracy on HellaSwag

Models trained on filtered data reach performance targets faster

Filtering reduces data volume by up to 25% without performance loss

Abstract

Data quality is crucial for training Large Language Models (LLMs). Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, we introduce an LLM-based line-level filtering method to enhance training data quality. We use GPT-4o mini to label a 20,000-document sample from FineWeb at the line level, allowing the model to create descriptive labels for low-quality lines. These labels are grouped into nine main categories, and we train a DeBERTa-v3 classifier to scale the filtering to a 10B-token subset of FineWeb. To test the impact of our filtering, we train GPT-2 models on both the original and the filtered datasets. The results show that models trained on the filtered data achieve higher accuracy on the HellaSwag benchmark and reach their performance targets faster, even with up to 25\% less data. This demonstrates that LLM-based line-level filtering can significantly improve data quality and training efficiency for LLMs. We release our quality-annotated dataset, FinerWeb-10BT, and the codebase to support further work in this area.