FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training

TL;DR

FIRE is a scalable framework that combines multiple data quality signals to improve data selection for pretraining large language models, leading to better performance with less data.

Contribution

We introduce FIRE, a novel method that integrates diverse quality ratings into a unified assessment, enhancing data selection for pretraining LLMs.

Findings

FIRE outperforms existing data selection methods.

FIRE achieves comparable performance with less than 37.5% of training data.

FIRE significantly boosts downstream task performance.

Abstract

Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signals, limiting their ability to evaluate data quality comprehensively. In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions. FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point. Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points. Extensive experiments show that FIRE outperforms other data selection methods and significantly boosts pretrained model performance across a wide range of downstream tasks, while requiring less than 37.5\% of the training data needed by the Random baseline to reach the target performance.