Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG

TL;DR

This paper introduces Synthetic Mixed Training, a method combining synthetic questions and documents to significantly improve language model knowledge acquisition beyond RAG, with demonstrated gains on multiple benchmarks.

Contribution

It presents Synthetic Mixed Training and Focal Rewriting techniques that enable models to outperform RAG by leveraging complementary synthetic data signals and question-conditioned document generation.

Findings

Outperforms RAG by 2.6% on QuaLITY benchmark.

Achieves 4.4% relative improvement over RAG with Llama 8B.

Outperforms RAG in five of six benchmark settings.

Abstract

Synthetic data augmentation helps language models learn new knowledge in data-constrained domains. However, naively scaling existing synthetic data methods by training on more synthetic tokens or using stronger generators yields diminishing returns below the performance of RAG. To break the RAG ceiling, we introduce Synthetic Mixed Training, which combines synthetic QAs and synthetic documents. This leverages their complementary training signals, and enables log-linear improvements as both synthetic data volume and generator strength increase. This allows the model to outperform RAG by a 2.6% relative gain on QuaLITY, a long-document reading comprehension benchmark. In addition, we introduce Focal Rewriting, a simple technique for synthetic document generation that explicitly conditions document generation on specific questions, improving the diversity of synthetic documents and yielding a steeper log-linear scaling curve. On QuaLITY, our final recipe trains a Llama 8B model that outperforms RAG by 4.4% relatively. Across models and benchmarks (QuaLITY, LongHealth, FinanceBench), our training enables models to beat RAG in five of six settings, outperforms by 2.6%, and achieves a 9.1% gain when combined with RAG.