Data-efficient pre-training by scaling synthetic megadocs

TL;DR

This paper demonstrates that synthetic data augmentation, especially using megadoc construction, significantly improves loss scaling and data efficiency in pre-training models, benefiting from increased synthetic data generation.

Contribution

It introduces novel methods for synthetic data generation, including megadoc construction, that enhance loss scaling and data efficiency in pre-training models.

Findings

Synthetic data improves validation loss despite distribution differences.

Megadoc construction increases data efficiency from 1.48x to 1.80x.

Megadocs outperform simple rephrasing, especially with more synthetic data.

Abstract

Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near $1.48\times$ data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from $1.48\times$ to $1.80\times$ at $32$ generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.