DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing

TL;DR

DeepSynth-Eval introduces an objective benchmark for evaluating how well large language models synthesize and organize information in long-form reports, addressing a key challenge in autonomous research agents.

Contribution

It presents a novel, fine-grained evaluation protocol and benchmark for assessing information consolidation in LLMs, using high-quality survey papers as gold standards.

Findings

Multi-reference synthesis remains challenging for LLMs.

Plan-and-write workflows outperform single-turn generation.

Structured evaluation reduces hallucinations and improves coherence.

Abstract

The evolution of Large Language Models (LLMs) towards autonomous agents has catalyzed progress in Deep Research. While retrieval capabilities are well-benchmarked, the post-retrieval synthesis stage--where agents must digest massive amounts of context and consolidate fragmented evidence into coherent, long-form reports--remains under-evaluated due to the subjectivity of open-ended writing. To bridge this gap, we introduce DeepSynth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities. We leverage high-quality survey papers as gold standards, reverse-engineering research requests and constructing "Oracle Contexts" from their bibliographies to isolate synthesis from retrieval noise. We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization), transforming subjective judgment into verifiable metrics. Experiments across 96 tasks reveal that synthesizing information from hundreds of references remains a significant challenge. Our results demonstrate that agentic plan-and-write workflows significantly outperform single-turn generation, effectively reducing hallucinations and improving adherence to complex structural constraints.