Escaping Collapse: The Strength of Weak Data for Large Language Model Training

TL;DR

This paper develops a theoretical framework inspired by boosting to determine the amount of data curation needed for continual improvement in large language model training with synthetic data, validated by experiments.

Contribution

It introduces a formal analysis of data curation in synthetic data training for LLMs, connecting boosting techniques to recent methods and proposing dynamic focusing strategies.

Findings

Proper curation prevents performance collapse

Focusing on challenging examples improves results

Theoretical insights align with experimental validation

Abstract

Synthetically-generated data plays an increasingly larger role in training large language models. However, while synthetic data has been found to be useful, studies have also shown that without proper curation it can cause LLM performance to plateau, or even "collapse", after many training iterations. In this paper, we formalize this question and develop a theoretical framework to investigate how much curation is needed in order to ensure that LLM performance continually improves. Our analysis is inspired by boosting, a classic machine learning technique that leverages a very weak learning algorithm to produce an arbitrarily good classifier. The approach we analyze subsumes many recently proposed methods for training LLMs on synthetic data, and thus our analysis sheds light on why they are successful, and also suggests opportunities for future improvement. We present experiments that validate our theory, and show that dynamically focusing labeling resources on the most challenging examples -- in much the same way that boosting focuses the efforts of the weak learner -- leads to improved performance.