Reformulation for Pretraining Data Augmentation

TL;DR

This paper introduces MGA, a scalable data augmentation method that reformulates existing text corpora to reduce repetition and improve large language model training efficiency, demonstrated on a 770 billion token dataset.

Contribution

We propose MGA, a novel reformulation technique for data augmentation that mitigates repetition issues and enhances large language model scaling, supported by the creation of the MGACorpus.

Findings

MGA outperforms data repetition and upsampling methods in scaling scenarios.

The MGACorpus contains 770 billion tokens for training.

Prompt engineering influences generation quality and evaluation metrics.

Abstract

Despite the impressive capabilities of large language models across various tasks, their continued scaling is severely hampered not only by data scarcity but also by the performance degradation associated with excessive data repetition during training. To overcome this critical bottleneck, we propose the Massive Genre-Audience(MGA) reformulation method, a lightweight and scalable data augmentation technique inspired by synthetic data methodologies. MGA systematically reformulates existing corpora into diverse, contextually-rich variations to mitigate the negative effects of repetition, and we introduce this approach along with the resulting 770 billion token MGACorpus in this work. We experimentally validate its core benefit by demonstrating superior performance against data repetition and upsampling in scaling scenarios (up to 13B parameters). Furthermore, comprehensive analysis investigates the role of prompt engineering in generation quality and reveals nuances in evaluating model capabilities using standard loss metrics. Our work shows that MGA provides a reliable pathway to substantially augment training datasets, effectively alleviating repetition bottlenecks and enabling more efficient scaling of large language models.