Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification

TL;DR

This paper investigates the use of verification methods to prevent model collapse when training large language models on synthesized data, providing theoretical insights and practical experiments demonstrating effectiveness.

Contribution

It introduces a theoretical framework for using verifiers to select high-quality synthesized data and demonstrates their effectiveness in preventing model collapse in practical tasks.

Findings

Verifiers can effectively prevent model collapse with synthesized data.

Proposed proxy measure correlates strongly with model performance.

Experimental results show improved model stability using verification.

Abstract

Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation. This raises concerns about \emph{model collapse}, a drop in model performance when their training sets include generated data. Considering that it is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with two practical tasks -- computing matrix eigenvalues with transformers and news summarization with LLMs -- which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.