Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory

TL;DR

This paper uses random matrix theory to analyze how synthetic data quality and pruning strategies affect the performance of classifiers trained on mixed real and synthetic data, revealing conditions for improvement and phase transitions.

Contribution

It extends previous analyses by applying random matrix theory to high-dimensional mixed data, providing new insights into synthetic data quality and verification effects.

Findings

Synthetic data can enhance classifier performance under certain conditions.

A smooth phase transition in label noise contrasts with previous sharp limits.

Theoretical results are validated with experiments on toy and large language models.

Abstract

Synthetic data has gained attention for training large language models, but poor-quality data can harm performance (see, e.g., Shumailov et al. (2023); Seddik et al. (2024)). A potential solution is data pruning, which retains only high-quality data based on a score function (human or machine feedback). Previous work Feng et al. (2024) analyzed models trained on synthetic data as sample size increases. We extend this by using random matrix theory to derive the performance of a binary classifier trained on a mix of real and pruned synthetic data in a high dimensional setting. Our findings identify conditions where synthetic data could improve performance, focusing on the quality of the generative model and verification strategy. We also show a smooth phase transition in synthetic label noise, contrasting with prior sharp behavior in infinite sample limits. Experiments with toy models and large language models validate our theoretical results.