WRAP++: Web discoveRy Amplified Pretraining

TL;DR

WRAP++ enhances large language model pretraining by discovering and synthesizing cross-document relationships from web hyperlinks, significantly amplifying training data with relational knowledge beyond single documents.

Contribution

It introduces a novel method for discovering cross-document relationships and synthesizing joint QA, greatly expanding training data for improved LLM knowledge acquisition.

Findings

Amplifies ~8.4B tokens to 80B tokens of cross-document QA data.

Models trained with WRAP++ outperform single-document approaches.

Cross-document synthesis yields better performance and scaling gains.

Abstract

Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowledge, missing cross-document relationships and leaving facts with limited associative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence relational motifs including dual-links and co-mentions, and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ~8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sustained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification.