SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training

TL;DR

This paper introduces SoftDedup, a data reweighting technique that reduces duplication impact in LLM pre-training, improving efficiency and downstream performance without losing valuable information.

Contribution

The paper presents a novel data commonness metric and a soft deduplication method that enhances training efficiency and downstream accuracy in language models.

Findings

Achieves at least 26% reduction in training steps for comparable perplexity.

Improves few-shot downstream accuracy by 1.77%.

Effective even on deduplicated datasets.

Abstract

The effectiveness of large language models (LLMs) is often hindered by duplicated data in their extensive pre-training datasets. Current approaches primarily focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication. To address this, we propose a soft deduplication method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. Central to our approach is the concept of "data commonness", a metric we introduce to quantify the degree of duplication by measuring the occurrence probabilities of samples using an n-gram model. Empirical analysis shows that this method significantly improves training efficiency, achieving comparable perplexity scores with at least a 26% reduction in required training steps. Additionally, it enhances average few-shot downstream accuracy by 1.77% when trained for an equivalent duration. Importantly, this approach consistently improves performance, even on rigorously deduplicated datasets, indicating its potential to complement existing methods and become a standard pre-training process for LLMs.