DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis

TL;DR

DeepScholar-bench is a novel live benchmark and automated evaluation framework designed to assess AI systems' ability to perform complex research synthesis tasks, such as generating related work sections from recent scientific papers.

Contribution

The paper introduces DeepScholar-bench, a comprehensive benchmark with an open-source reference pipeline for evaluating generative research synthesis systems on real-world tasks.

Findings

No existing system surpasses 31% performance across all metrics

DeepScholar-bench reveals the significant challenge of automated research synthesis

Provides a new foundation for advancing AI in scientific knowledge synthesis

Abstract

The ability to research and synthesize knowledge is central to human expertise and progress. A new class of AI systems--designed for generative research synthesis--aims to automate this process by retrieving information from the live web and producing long-form, cited reports. Yet, evaluating such systems remains an open challenge: existing question-answering benchmarks focus on short, factual answers, while expert-curated datasets risk staleness and data contamination. Neither captures the complexity and evolving nature of real research synthesis tasks. We introduce DeepScholar-bench, a live benchmark and automated evaluation framework for generative research synthesis. DeepScholar-bench draws queries and human-written exemplars from recent, high-quality ArXiv papers and evaluates a real synthesis task: generating a related work section by retrieving, synthesizing, and citing prior work. Our automated framework holistically measures performance across three key dimensions--knowledge synthesis, retrieval quality, and verifiability. To further future work, we also contribute DeepScholar-ref, a simple, open-source reference pipeline, which is implemented on the LOTUS framework and provides a strong baseline. Using DeepScholar-bench, we systematically evaluate prior open-source systems, search agents with strong models, OpenAI's DeepResearch, and DeepScholar-ref. We find DeepScholar-bench is far from saturated: no system surpasses a geometric mean of $31\%$ across all metrics. These results highlight both the difficulty and importance of DeepScholar-bench as a foundation for advancing AI systems capable of generative research synthesis. We make our benchmark code and data available at https://github.com/guestrin-lab/deepscholar-bench.

Tables

Table 1. Table 1: Summary of Evaluation Metrics.

Metric	Description
Knowledge Synthesis
Organization & Coherency	assesses organization and coherence of system answer
Nugget Coverage	assesses the answer’s coverage of essential facts
Retrieval Quality
Relevance Rate	measures avg. relevance among all referenced sources
Document Importance	measures how notable referenced sources are, using citation counts
Reference Coverage	assesses the referenced set’s coverage of key, important references
Verifiability
Citation Precision	measures percent of cited sources that support their accompanying claim
Claim Coverage	measures percent of claims that are fully supported by cited sources

Table 2. Table 2 : Main Results. The best baseline is shown in bold and the second-best baseline is underlined. ∗ indicates that the best baseline is statistically significantly better than the second-best baseline under a paired two-tailed t-test with p < 0.05 p<0.05 .

	Knowledge Synthesis		Retrieval Quality			Verifiability		Geo. Mean
	Org.	Nug. Cov.	Rel. Rate	Ref Cov.	Doc Imp.	Cite-P	Claim Cov ($w=1$)
Human-written Exemplars
Human-written Exemplars	.500	1.000	.585	1.000	1.000	.90011footnotemark: 1	.85011footnotemark: 1	.78211footnotemark: 1
Open Source Research Systems
DeepResearcher (Llama-4)	.206	.230	.385	.047	.008	.312	.396	.137
STORM (Llama-4)	.119	.183	.218	.003	.006	.238	.586	.073
OpenScholar (Llama-4)	.309	.278	.017	.008	.013	.010	.138	.042
Search Agents
Search Agent (Llama-4)	.151	.193	.445	.060	.009	.316	.368	.135
Search Agent (GPT-4.1)	.556	.265	.490	.050	.009	.498	.470	.186
Search Agent (o3)	.849	.348	.610	.165	.026	.425	.495	.287
Search Agent (Claude)	.698	.307	.583	.131	.008	.701	.760	.256
Search Agent (Gemini)	.706	.277	.583	.061	.010	.415	.398	.196
Commercial Systems
OpenAI DeepResearch	.857	.392∗	.629	.187∗	.124∗	.399	.138	.309∗
DeepScholar Reference Pipeline
DeepScholar-ref (Llama-4)	.206	.241	.436	.103	.008	.674	.851	.195
DeepScholar-ref (GPT-4.1)	.809	.348	.590	.166	.008	.788	.899	.285
DeepScholar-ref (GPT-4.1, o3)	.857	.384	.645	.167	.007	.824	.760	.285
DeepScholar-ref (GPT-4.1, Claude)	.698	.307	.610	.152	.009	.944∗	.895	.286
DeepScholar-ref (GPT-4.1, Gemini)	.770	.331	.590	.181	.006	.904	.937∗	.282

Table 3. Table 3 : Ablation Study Comparing The Effect of Different Retrieval APIs. The best baseline is shown in bold and the second-best baseline is underlined. ∗ indicates that the best baseline is statistically significantly better than the second-best baseline under a paired two-tailed t-test with p < 0.05 p<0.05 .

	Knowledge Synthesis		Retrieval Quality			Verifiability		Geo. Mean
	Org	Nug. Cov.	Rel. Rate	Ref Cov.	Doc Imp.	Cite-P	Claim Cov ($w=1$)
DeepScholar-ref (GPT-4.1, Claude)
arxiv.org Retrieval	.698	.307	.610	.152	.009	.944	.895	.286
parallel.ai Retrieval	.865	.444	.675	.160	.017	.846	.781	.334
taviliy.com Retrieval	.929∗	.327	.550	.070	.015	.711	.578	.258
Oracle Retrieval (arxiv.org)	.782	.487	.686	1.000∗	1.000	.955	.899∗	.808
Oracle Retrieval (All)	.778	.528∗	.680	1.000∗	.822	.941	.828	.782
DeepScholar-ref (Llama-4)
arxiv.org Retrieval	.206	.241	.436	.103	.008	.674	.851	.195
parallel.ai Retrieval	.246	.265	.559	.114	.015	.223	.543	.186
taviliy.com Retrieval	.111	.229	.532	.030	.016	.442	.676	.153
Oracle Retrieval (arxiv.org)	.202	.316	.681	1.000∗	1.000	.658	.868	.590
Oracle Retrieval (All)	.198	.350	.693	1.000∗	.822	.796∗	.890	.600

Table 4. Table 4 : Confusion matrices comparing human and LLM judgments on organization, Nugget importance judgments, and Reference-importance judgments, where in each table rows and columns represent human and LLM judgments.

Organization
Human / LLM	Reference	Generated	Tie
Reference	14.29%	0%	14.29%
Generated	0%	57.14%	14.29%
Tie	0%	0%	0%

Organization
Human / LLM	Reference	Generated	Tie
Reference	14.29%	0%	14.29%
Generated	0%	57.14%	14.29%
Tie	0%	0%	0%

Nugget Importance
Human / LLM	Vital	Okay
Vital	58.33%	8.33%
Okay	8.33%	25.00%
Irrelevant	0%	0%

Reference Importance
Human / LLM	Not Imp.	Imp.
Not Imp.	40.2%	9.8%
Imp.	24.2%	25.7%

Table 8. Table 5 : Title and ArXiv IDs of papers included in DeepScholar-June-2025.

#	ArXiv ID	Title
0	2506.02838v1	TaxAgent: How Large Language Model Designs Fiscal Policy
1	2506.02634v1	KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
2	2506.00958v1	Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues
3	2506.00832v1	Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models
4	2506.00418v1	Dual Debiasing for Noisy In-Context Learning for Text Generation
5	2505.24754v1	Don’t Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation
6	2505.24575v1	NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization
7	2506.00085v1	COSMIC: Generalized Refusal Direction Identification in LLM Activations
8	2505.23996v1	Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs
9	2505.23353v1	Synthetic Generation and Latent Projection Denoising of Rim Lesions in Multiple Sclerosis
10	2505.22757v1	Pre-Training Curriculum for Multi-Token Prediction in Language Models
11	2506.02853v1	Learning Pyramid-structured Long-range Dependencies for 3D Human Pose Estimation
12	2506.02547v1	Probabilistic Online Event Downsampling
13	2506.01071v1	Aligned Contrastive Loss for Long-Tailed Recognition
14	2506.01037v1	Self-supervised ControlNet with Spatio-Temporal Mamba for Real-world Video Super-resolution
15	2506.00434v1	Efficient 3D Brain Tumor Segmentation with Axial-Coronal-Sagittal Embedding
16	2506.00333v1	Test-time Vocabulary Adaptation for Language-driven Object Detection
17	2505.24443v1	Diversify and Conquer: Open-set Disagreement for Robust Semi-supervised Learning with Outliers
18	2505.24334v1	KairosAD: A SAM-Based Model for Industrial Anomaly Detection on Embedded Devices
19	2505.23290v1	Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation
20	2505.23180v1	Proximal Algorithm Unrolling: Flexible and Efficient Reconstruction Networks for Single-Pixel Imaging
21	2505.22616v1	PS4PRO: Pixel-to-pixel Supervision for Photorealistic Rendering and Optimization
22	2505.22458v1	Universal Domain Adaptation for Semantic Segmentation
23	2505.22427v1	RC-AutoCalib: An End-to-End Radar-Camera Automatic Calibration Network
24	2505.22167v1	Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers
25	2505.22552v1	ClaimPKG: Enhancing Claim Verification via Pseudo-Subgraph Generation with Lightweight Specialized LLM
26	2504.21752v1	VDDP: Verifiable Distributed Differential Privacy under the Client-Server-Verifier Setup
27	2504.21282v1	Birdie: Natural Language-Driven Table Discovery Using Differentiable Search Index
28	2504.17448v1	CHASe: Client Heterogeneity-Aware Data Selection for Effective Federated Active Learning
29	2504.14861v1	Stitching Inner Product and Euclidean Metrics for Topology-aware Maximum Inner Product Search
30	2504.06975v1	AWDIT: An Optimal Weak Database Isolation Tester
31	2506.01833v1	SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model
32	2506.00382v1	Spectral Insights into Data-Oblivious Critical Layers in Large Language Models
33	2506.00205v1	Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective
34	2505.24835v1	Timing is important: Risk-aware Fund Allocation based on Time-Series Forecasting
35	2505.24203v1	Aligning Protein Conformation Ensemble Generation with Physical Feedback
36	2506.02847v1	CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
37	2505.22194v1	Refining Datapath for Microscaling ViTs
38	2505.11554v1	Multi-Objective Memory Bandwidth Regulation and Cache Partitioning for Multicore Real-Time Systems
39	2505.08071v1	NMP-PaK: Near-Memory Processing Acceleration of Scalable De Novo Genome Assembly
40	2504.06211v1	Need for zkSpeed: Accelerating HyperPlonk for Zero-Knowledge Proofs
41	2504.19283v1	Efficient Serverless Cold Start: Reducing Library Loading Overhead by Profile-guided Optimization
42	2504.11007v1	Kubernetes in the Cloud vs. Bare Metal: A Comparative Study of Network Costs
43	2504.09307v1	Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training
44	2506.02750v1	Learning Binarized Representations with Pseudo-positive Sample Enhancement for Efficient Graph Collaborative Filtering
45	2505.23452v1	What About Emotions? Guiding Fine-Grained Emotion Extraction from Mobile App Reviews
46	2505.21811v1	Revisiting Self-attention for Cross-domain Sequential Recommendation
47	2505.20227v1	Measure Domain’s Gap: A Similar Domain Selection Principle for Multi-Domain Recommendation
48	2505.19356v1	Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval
49	2505.19307v1	Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization
50	2505.17507v1	Benchmarking Recommendation, Classification, and Tracing Based on Hugging Face Knowledge Graph
51	2505.12791v1	Unlearning for Federated Online Learning to Rank: A Reproducibility Study
52	2505.07166v1	Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition
53	2505.03484v1	STAR-Rec: Making Peace with Length Variance and Pattern Diversity in Sequential Recommendation
54	2505.00552v1	Graph Spectral Filtering with Chebyshev Interpolation for Recommendation
55	2504.20458v1	Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated User
56	2504.18383v1	Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation
57	2504.17519v1	Replication and Exploration of Generative Retrieval over Dynamic Corpora
58	2504.15849v1	NLCTables: A Dataset for Marrying Natural Language Conditions with Table Discovery
59	2504.14991v1	Understanding Accuracy-Fairness Trade-offs in Re-ranking through Elasticity in Economics
60	2504.14243v1	Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems
61	2504.12900v1	FashionDPO:Fine-tune Fashion Outfit Generation Model using Direct Preference Optimization
62	2504.09935v1	Constrained Auto-Regressive Decoding Constrains Generative Retrieval

Table 9. Table 6 : Report Statistics.

	Report Length			Citations		Cost
	Chars	Words	Sentences	# Unique Refs	# Inline Citations	Latency (s)	Dollar Cost (USD)
Human-written Exemplars
Human-written Exemplars	4381	497	28	23	27	N/A	N/A
Open Source Research Systems
DeepResearcher (Llama-4)	2573	319	35	8	7	31	0.00
STORM (Llama-4)	2766	381	31	18	21	162	0.00
OpenScholar (Llama-4)	3513	483	26	9	19	87	0.00
Search Agents
Search Agent (Llama-4)	1968	258	16	9	5	20	0.00
Search Agent (GPT-4.1)	3168	404	16	10	61	39	0.07
Search Agent (o3)	3844	501	24	11	16	263	0.15
Search Agent (Claude)	3977	499	27	13	8	147	1.36
Search Agent (Gemini)	2810	395	19	6	8	442	0.11
Commercial Systems
OpenAI DeepResearch	6577	864	74	17	6	630	5.02
DeepScholar Reference Pipeline
DeepScholar-ref (Llama-4)	3499	360	53	19	35	313	0.00
DeepScholar-ref (GPT-4.1)	7863	735	115	20	89	234	1.66
DeepScholar-ref (GPT-4.1, o3)	5726	617	70	17	42	276	1.15
DeepScholar-ref (GPT-4.1, Claude)	5855	618	72	17	40	334	1.23
DeepScholar-ref (GPT-4.1, Gemini)	5623	570	86	24	63	349	1.29

Table 10. Table 7 : Statistics Related to Evaluation Metrics.

	Avg. value over human-written exemplars	Relevant Metric
# Important References from ArXiv.org	11.47	Ref. Cov.
Median number of citations per reference from ArXiv.org	647.5	Doc. Imp.

Table 11. Table 8 : Performance of selected systems on DeepScholar-Nov-2025 (200 queries across > > 75 arXiv subject areas). We report the same metrics as in Table  2 .

Model	Org.	Nug. Cov.	Rel. Rate	Ref Cov.	Doc Imp.	Cite-P	Claim Cov.
DeepScholar-ref (Llama-4-Scout)	0.120	0.358	0.395	0.072	0.061	0.178	0.581
DeepScholar-ref (GPT-4.1 + o3)	0.578	0.479	0.568	0.087	0.054	0.563	0.578
Search Agent (Llama-4-Scout)	0.108	0.252	0.179	0.034	0.124	0.189	0.314
Search Agent (o3)	0.608	0.480	0.623	0.078	0.763	0.475	0.452

Table 12. Table 9 : Pearson correlations between system-level scores under different query formulations (abstract, Key Idea , and RQ ). Each entry is computed over the scores of our main baselines. Correlations marked with ∗ are statistically significant under a permutation-based paired test with threshold p < 0.05 p{<}0.05 .

Query pair	Org.	Nug.	Cov. Rel. Rate	Ref Cov	Doc Imp	Cite-P	Claim Cov ($w=1$)
Abstract vs. Key Idea	0.997∗	0.980∗	0.979∗	0.992∗	0.992	0.589	0.841∗
Abstract vs. RQ	0.988∗	0.979∗	0.772∗	0.951∗	0.707	0.766∗	0.877∗
Key Idea vs. RQ	0.991∗	0.983∗	0.836∗	0.958∗	0.766∗	0.942∗	0.981∗

Table 13. Table 10 : Manual Validation of LLM-based Evaluation

Evaluation Metric	LLM-Classified Labels	Human-Agreement Score with LLM
Organization	Pairwise Comparison (Lose / Tie / Win)	$78\%$
Nugget Coverage	Nugget Importance (Vital / Non-vital)	$72\%$
Nugget Coverage	Nugget Coverage (Supported / Partially Supp. / Not Supp.)	$70\%$
Retrieval Relevance Rate	Graded Relevance (0/1/2)	$70\%$
Reference Coverage	Reference Importance (Not Imp./ Imp.)	$82\%$
Document Importance	N/A	N/A
Citation Precision	Entailment (Entailed / Not Entailed)	$80\%$
Claim Coverage	Entailment (Entailed / Not Entailed)	$80\%$

Table 14. Table 11 : Pearson correlations between system-level scores produced by different LLM judges across five baselines for each metric. Statistical significance is assessed using a permutation-based paired t-test with threshold p < 0.05 p{<}0.05 ; correlations with p < 0.05 p{<}0.05 are marked with ∗ .

Judge pair	Org.	Nug.	Cov. Rel. Rate	Ref. Cov.	Cite-P	Claim Cov ($w=1$)
GPT-4o vs. Llama-4	0.985∗	0.413	0.941	0.975∗	0.817∗	0.958∗
GPT-4o vs. DeepSeek	0.966∗	0.920∗	0.980∗	0.990∗	0.843∗	0.996∗
DeepSeek vs. Llama-4	0.991∗	0.668	0.942∗	0.994∗	0.986	0.963∗

Equations

$$RR(S)=\frac{1}{2|S|}\sum _{s\in S}Rel(s).$$

$$RC(S,E)=\frac{1}{|E|}\sum _{s\in S}I[s\in E].$$

$$DI(S,S^{\ast })=min(\frac{\mathrm{median}\{num-cites(s)|s\in S\}}{\mathrm{median}\{num-cites(s^{\ast })|s^{\ast }\in S^{\ast }\}},1),$$

Full text

1 Introduction

A core foundation of human knowledge and innovation is the ability of human experts to research and synthesize known facts and new findings, enabling others to comprehend, verify and build upon prior work. Recently, systems for generative research synthesis have emerged, promising to automate tasks that produce long-form outputs (e.g., multi-page reports), which traditionally demand hours of literature searching, reading and writing by human experts. These offerings include commercial ones—from OpenAI [33], Gemini [13], Anthropic [2], Grok [59], and Perplexity [38]—as well as open-source methods, such as STORM [43], DeepResearcher [65], and OpenScholar [6]. Existing systems demonstrate promising performance on factuality and question-answering benchmarks [55, 20, 31, 56], pushing the frontier of AI capabilities.

Yet, as this new class of systems emerges, a key question remains: how should we benchmark and evaluate generative research synthesis? The progress of these systems requires benchmarks that carefully evaluate their critical capabilities—specifically, three core functions: (1) retrieval, typically from a large, complex, and constantly-evolving corpus, such as the live web, to collect key information (2) knowledge synthesis, to generate coherent, long-form answers that surface key facts, integrating general knowledge and findings from many retrieved sources, and (3) verifiability, providing citations that allow readers to trace each stated claim in the synthesized answer to a reputable source from the retrieved set. The ideal benchmark must holistically evaluate across all three of these dimensions, while providing a realistic and challenging research synthesis task.

Unfortunately, existing benchmarks fall short of these goals. Many prior works evaluate generative research synthesis systems using existing question answering benchmarks, which do not reflect realistic research synthesis tasks and instead focus on questions with short-form, easily-verifiable answers, making them severely limited for this setting [56, 55, 31, 20, 58, 53, 18, 61, 19, 21, 15, 49, 23]. These question-answering benchmarks do not capture the complexity of long-form answers synthesized from many sources, a key component of research synthesis. To address this limitation, several recent works instead leverage expert-curated datasets with open-ended research questions and exemplar answers [6, 66, 62, 60, 8, 45, 16]. Unfortunately, these benchmarks quickly become stale and outdated as new information emerges. Furthermore, these datasets risk data contamination as new models are trained on snapshots of the web, including public datasets. The prohibitive expense of curating, maintaining, and updating expert-curated benchmarks further limits their utility towards realistic, scalable evaluation.

In this work, we introduce DeepScholar-bench, a live benchmark and holistic, automated evaluation framework designed to evaluate generative research synthesis. DeepScholar-bench draws queries from recent, high-quality ArXiv papers and focuses on a real research synthesis task: generating the related work sections of a paper by retrieving, synthesizing, and citing prior research. We plan to provide a live benchmark, releasing updated research queries every month, and practitioners can also run our automated date pipeline to create their own dataset instantiations. Further, we develop an automated evaluation framework that leverages human-written related works extracted from each ArXiv paper and holistically assesses performance across three key dimensions— knowledge synthesis, retrieval quality, and verifiability—using metrics that show strong agreement with human judgments. To promote future work, we also develop DeepScholar-ref, a simple open-source reference pipeline for generative research synthesis implemented on the LOTUS framework [28, 37].

Using the DeepScholar-bench framework, we systematically evaluate the performance of existing systems, including open-source research synthesis systems, search agents with strong proprietary models, OpenAI DeepResearch, and DeepScholar-ref. We find that all of these existing methods exhibit significant opportunity for improvement, with no system surpassing a geometric mean of $31\%$ across all metrics. Furthermore, on several key metrics, including Nugget Coverage, Reference Coverage and Document Importance, each evaluated method’s performance remains well below $40\%$, reflecting the inherent difficulty of the DeepScholar-bench task, which requires systems to navigate the live web, reasoning about the relevance and importance of documents as well as surfacing key facts into a cohesive final answer. Notably, OpenAI’s DeepResearch offers strong performance relative to other baselines, outperforming many prior methods on knowledge synthesis and retrieval quality, with scores of $39.2\%$ on Nugget Coverage, $18.7\%$ on Reference Coverage and $12.4\%$ on Document Importance; however, it struggles to provide strong verifiability relative to many other methods. We also find that DeepScholar-ref reference pipeline represents a strong open-source baseline offering competitive performance on most metrics and up to $6.3\times$ higher verifiability compared to OpenAI’s DeepResearch. Nevertheless, DeepScholar-bench remains far from saturated, representing exciting opportunities for further work. We hope that our benchmark framework and reference pipeline support the progress of new systems, and we believe that resolving DeepScholar-bench represents a critical milestone towards more capable AI systems.

Overall, our main contributions are the following:

• •

We propose DeepScholar-bench, a live benchmark dataset with real research synthesis tasks and an automated, holistic evaluation.

• •

We develop DeepScholar-ref, a simple open-source reference pipeline for generative research synthesis that attains competitive performance with open-source systems, search agents, and OpenAI’s DeepResearch across many metrics using the same models.

• •

We perform a systematic evaluation of existing baselines on DeepScholar-bench, finding significant opportunities for improvement, with no system surpassing a geometric mean of $31\%$ across all metrics.

2 The DeepScholar Dataset

We study the task of generating a related works section of an academic paper, a fundamental research synthesis task, for which we leverage human-written exemplars from our automated data pipeline to ground our evaluation. We source our dataset queries by scraping ArXiv papers [5] accepted at academic conferences, and we formalize our dataset task as follows: given a paper’s description $d$, the goal is to retrieve a set of relevant sources $S$ and generate a related works sections $W$ by synthesizing and citing the retrieved documents. We briefly overview our automated data collection framework (Section 2.1) and describe the dataset instantiation (Section 2.2) used in our evaluation (Section 5). Appendix A.1 provides additional details.

2.1 Automated Data Collection Framework

Our automated data collection framework aims to achieve the following design goals:

Inclusion of diverse paper topics across a wide variety of research domains. 2. 2.

Focus on recent research papers, both to provide realistic, timely benchmark queries and to prevent data contamination when benchmarking models trained on snapshots of the web. 3. 3.

Control for quality of the scraped ArXiv papers and extracted data, focusing on peer-reviewed manuscripts that are accepted at academic conferences.

Our data pipeline scrapes papers, filters them, and extracts content to construct datasets that include metadata about each ArXiv paper (e.g., the title, abstract, and link), the papers’ related work section, and a citation list of references found in the papers’ related work sections. Our scraper begins by loading papers from a configured set of ArXiv domains (e.g., cs.ML) and configured publication-date range. To avoid possible data contamination arising from multiple ArXiv versions, we keep only v1 ArXiv papers. To control for paper quality, our pipeline then optionally filters papers to keep only those listed as “accepted” or “published” at a conference based on the ArXiv metadata. We also exclude papers that do not have an explicit “Related Work” section and .bib file, containing well-formatted bibliography entries. Our pipeline then processes each paper to extract the Related Works section from both the LaTex files and PDF file, if available. We clean the extracted LaTex section to remove any labels and comments. We also extract all citations found in the related work section and we use the ArXiv and OpenAlex APIs [35] to recover more detailed information, such as abstracts, authors, and links for both ArXiv and non-ArXiv references.

2.2 Dataset Description and Statistics

The dataset instantiation, DeepScholar-June-2025 used in our evaluation (Section 5) configures a publication date range between April and June 2025, following the April 5th, 2025 release date of Llama-4 models [27], the main open-source model we benchmark. This instantiation scrapes papers from a diverse set of 18 distinct ArXiv domains—including, cs.IR, cs.CV, cs.AI, cs.CL, cs.LG, cs.DC, cs.DB, cs.AR, cs.SD, cs.CR, cs.ET, cs.GR, cs.PL, cs.SY, cs.OS, cs.PF, cs.SE, cs.MM—and selects papers marked as accepted at a conference. We additionally exclude papers with related works sections longer than 1,000 words to control for cost. Our final dataset includes 63 ArXiv papers, each providing a single query and an extracted expert-written exemplar for our evaluation. We make our scripts available to allow others to configure different datasets, and we plan to release monthly datasets of recent queries. Our experiments leverage the abstract of each paper as the paper’s description $d$, provided to each baseline system as context within the query. We analyze the human-written exemplars from our dataset, and we find that, on average, each related work section contains $23$ unique references, and we find over $63\%$ of all cited references on ArXiv. We provide additional experiments in Appendix A.3.2 on a more recent, expanded dataset DeepScholar-Nov-2025, which contains 200 queries from over 75 distinct arXiv subject areas, spanning Computer Science, Physics, Quantitative Biology, Economics, and Quantitative Finance. Our results on this dataset confirm the generalization of our benchmark and main experimental conclusions.

3 The DeepScholar Evaluation Framework

Evaluating research synthesis is inherently challenging due to the task’s complexity, and the lack of a simple, "ground truth" notion of correctness, permitting many possible answers. To address these challenges, we propose a holistic, automated evaluation that assesses 7 fine-grained metrics across three key dimensions that reflect the core capabilities of research synthesis: knowledge synthesis (Section 3.1), retrieval quality (Section 3.2), and verifiability (Section 3.3). We overview our evaluation framework in this section, and provide further details and analysis of our metrics in Appendix A.4, as well as manual validation of our LLM-based metrics in Appendix A.3.

3.1 Knowledge Synthesis

Knowledge synthesis reflects the ability of a system to generate an effective final report which surfaces key facts and information into a coherent writeup. We evaluate both the overall Organization and Coherency of each system report, as well as its factual content according to its Nugget Coverage.

Organization and Coherency. We use an LLM-as-a-judge to assess organization and coherence and perform pairwise comparisons between the system generated report and the human-written exemplar from the dataset. We permute each evaluated pair of reports to avoid position bias [24], and we report the win-rate of each baseline. This model-based evaluation provides scalability while also serving as a strong surrogate for human preferences [42, 25, 24, 4], which we validate in our experiments in Appendix A.3.

Nugget Coverage. To assess the quality of the information content of a generated report, we use a nugget-based evaluation. An information nugget is an essential fact or component relevant for an answer [39, 51, 50, 9, 41]. For our task, we generate nuggets from the human-written exemplar related-work section for each query, and for each generated report, we compute the nugget coverage score, the fraction of nuggets present in the answer, following the automated, LLM-based methodology of [39].

3.2 Retrieval Quality

A key component of generative research synthesis is retrieval over the live web, which differs substantially from traditional information retrieval evaluations [47, 32]. This setting lacks a closed corpus with gold labels — expert-written exemplars provide one reasonable reference set, but there may be many possible alternative sets that are likewise high-quality. To address these challenges, we evaluate three metrics of each generated report’s retrieved reference set: the relevance rate, reference coverage of key sources, and document importance.

Relevance Rate. We asses the relevance of each retrieved document, following the Cranfield model [52], which is standard in IR evaluations and considers relevance of individual documents given a query, independent of other documents. We use an LLM-as-a-judge approach to assign graded relevance scores to each retrieved source, following recent works [51, 9, 42, 48, 6]. Specifically, the LLM-judge assigns a relevance score, $Rel(s)$, from $0-2$ for each source, $s$, in the retrieved set $S$, and we compute the relevance rate of $S$ as:

[TABLE]

Reference Coverage. We introduce a metric to measure the reference coverage of each report’s retrieved set. A key challenge in measuring this value is in defining a core set of "important" sources that a good report should reference. To build this set, we take all references from the high-quality, human-written exemplar and label each as either "important" or "not-important", considering a "not-important" reference as one that could be omitted or substituted by a different source. For each generated report, we then report its reference coverage by dividing the number of retrieved important references by total number of important references. We follow the below formula, where $E$ is the set of "important" references from the human-written exemplar:

[TABLE]

Document Importance. While the above retrieval metrics assess topical relevance and coverage of key references, an ideal research synthesis system must also retrieve many notable and important sources. Exemplar human-written reports typically contain ample references of primary-sources and highly-cited academic publications. We compute the document importance of a retrieved set by considering the number of citations of each of its sources. Specifically, we consider the median number of citations over sources in $S$, compared to this the median number of citations over sources in the reference set, $S^{*}$, from the human-written exemplar, and set an upper-bound of 1, given by:

[TABLE]

where $\operatorname{num-cites}(s)$ is the number of citations for source $s$.

3.3 Verifiability

To evaluate the verifiability generated reports using the citation precision and claim coverage with LLM-based entailment evaluations, following prior work [11, 26].

Citation Precision. We measure sentence-level precision, where a citation is considered precise if the referenced source supports at least one claim made in the accompanying sentence. For a full report, we compute citation precision by averaging the precision of each citation.

Claim Coverage. Claim coverage of a report is computed by assigning a sentence-level score—of one if the sentence’s cited sources support all claims made in the sentence—and averaging all sentence-level scores in the report. We make two adaptations to the original definition of prior work [11, 57, 26] to tailor our metric to our long-form synthesis task. First, we relax the original claim coverage definition to consider a sliding window of sentences with supporting references Specifically, we assign a sentence-level coverage score of 1 if the sentence is fully supported by the sources cited either within the sentence or in a window of $w$ preceding or following sentences. Additionally, since our task query provides context describing a paper, we consider this context as an implicitly cited reference for each sentence.

4 DeepScholar-ref

We introduce DeepScholar-ref, an open-source reference pipeline designed for generative research synthesis. As Figure 2 shows, DeepScholar-ref takes a user’s query and iteratively generates web-search queries, summarizing search results in each round before generating new queries. The system then post-processes the search results leveraging a series of semantic operators [37] implemented on the LOTUS API [28]. This includes a semantic filtering step, which leverages an LLM to filter out irrelevant source documents, followed by a semantic top-k which performs an LLM-based ranking over the documents based on their relevance to the user query. Finally, we perform a semantic aggregation over the final source documents to generate the final report. We provide further details of each step of our reference pipeline in Appendix Section A.6.

5 Experimental Results

In this section, we evaluate recent state-of-the-art generative research systems as well as DeepScholar-ref on DeepScholar-bench. Overall, we find the following:

• •

Existing baselines for generative research synthesis, including strong open-source LLM systems, search agents, and commercial systems, demonstrate significant room for improvement across all three key dimensions: knowledge synthesis, retrieval quality and verifiability. Specifically, no system surpasses a geometric mean of $31\%$ across all metrics.

• •

DeepScholar-ref provides a strong baseline, consistently improving upon the performance of prior open-source systems and search agents, as well as achieving competitive performance and up to $6.3\times$ higher verifiability compared to OpenAI’s DeepResearch.

Experimental Setup. We benchmark open-source research systems, including DeepResearcher [65], STORM [43] and OpenScholar [6], search agents, with Llama-4-Scout-17B-16E-Instruct [30], GPT-4.1-2025-04-14 [34], o3-2025-04-16 [34], Claude-opus-4-20250514 [3], and Gemini-2.5-pro [14] models, OpenAI’s o3-deep-research [34], and DeepScholar-ref. For each benchmarked method, we control the retrieval corpus by allowing each system to access the Web only through the ArXiv API [5]. We additionally avoid possible information leakage during search by filtering out any search results that were published after the query paper’s publication date. We provide further details of our setup in Appendix A.2.

5.1 Main Results

Table 2 provides of each method’s performance on DeepScholar-June-2025. For each baseline, we report metrics averaged over all queries, and the geometric mean of all metrics. We also provide additional results in Appendix A.3—including metadata statistics of the generated reports (Table 6), statistics related to our evaluation metrics (Table 7), and manual validation of our LLM-based metrics (Table 10)—and example reports in Appendix A.7. We discuss key findings in detail below.

5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement.

From Table 2, we see that no system surpasses a geometric mean of .31 across all metrics, with OpenAI DeepResearch obtaining the highest geometric mean. Moreover, on several key metrics, including Nugget Coverage, Reference Coverage and Document Importance, each baseline’s performance remains below $.40$. This reflects the inherent difficulty of the generative research task provided by DeepScholar-bench, particularly the need for systems to navigate the live web, reasoning about coverage and importance of documents, before surfacing key information in long-form report.

We now analyze each evaluated dimension, comparing performance of the open-source research systems, search agents and commercial systems to the human-written exemplars. On Knowledge Synthesis, we see that OpenAI DeepResearch offers the best performance compared to all other prior methods on both Organization, with a score of $.857$, and Nugget Coverage, with a score of $.392$. OpenAI DeepResearch, as well as the o3, Claude and Gemini search agents achieve relatively high Organization scores compared to human-written exemplars. However, on Nugget Coverage all prior methods scores below $.40$. This demonstrates that while existing systems, especially those using state-of-the-art models, can generate well-organized and coherent summaries, they still struggle to surface key facts to answer the research query, a crucial capability for research synthesis tasks.

Turning our attention to the Retrieval Quality performance of prior methods, we once again find significant room for improvement. Once again, OpenAI DeepResearch offers the strongest performance among the other benchmarked prior methods on Relevance Rate, Reference Coverage and Document Importance, but still far from saturates performance. While it’s Relevance Rate shows strong performance, exceeding that of the human exemplars with a score of $.629$, it’s Reference Coverage and Document Importance scores remain exceedingly low: $.187$ and $.124$ respectively. This demonstrates that while state-of-the-art generative research synthesis systems excel at retrieving relevant sources, they still struggle to find comprehensive sets of notable sources, falling short compared to the ability of human experts.

Lastly, we analyze the Verifiability performance of prior methods. We see that OpenAI DeepResearch is outperformed on both Citation Precision and Claim Coverage by the search agents with GPT4.1, o3, Claude and Gemini models. The Claude search agent offers the highest Citation Precision, a score of $.701$ and Claim Coverage, a score of $.760$. Meanwhile, OpenAI’s DeepResearch as well as the all other prior methods are unable to achieve a Citation Precision score beyond $.50$ and a Claim Coverage score beyond $.60$. We also note that the human-written exemplars appear to exhibit rather low Citation Precision and Claim Coverage scores, however these scores are under-estimate the actual verifiability of human writing11footnotemark: 1. Overall, we see that prior LLM-based systems exhibit significant headroom for improvement.

5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis.

We compare the performance of DeepScholar-ref to OpenAI DeepResearch, search agents and open-source systems, finding that DeepScholar-ref provides a strong baseline with competitive performance against the other baselines across most metrics using the same or cheaper models. In comparison to OpenAI DeepResearch, DeepScholar-ref (GPT-4.1, o3) achieves a similar or higher Organization, Nugget Coverage, Relevance Rate, Reference Coverage, Citation Precision and Claim Coverage score. Notably, DeepScholar-ref achieves up to $6.3\times$ higher Verifiability scores but its Document Importance remain relatively low compared to OpenAI’s DeepResearch. In Appendix Table 6, we provide additional cost analysis, finding that DeepScholar-ref (GPT-4.1, o3) offers an efficient reference pipeline that is $4.3\times$ cheaper and $2.28\times$ faster than OpenAI DeepResearch.

Next, we compare the performance of DeepScholar-ref and search agents, finding DeepScholar-ref offers competitive and often stronger performance—specifically, averaged across the 5 baselines, using the same primary model, DeepScholar-ref increases Organization by $1.18\times$, Nugget Coverage by $1.17\times$, the Relevance Rate by $1.06\times$, Reference Coverage by $2.03\times$, Citation Precision by $1.83\times$ and Claim Coverage by $1.86\times$. Lastly, we compare DeepScholar-ref (Llama-4) to the open-source research systems, all run with the Llama-4. We see that the prior open-source research systems exhibit trade-offs among the Knowledge Synthesis, Retrieval Quality and Verifiability dimensions. Compared to the best-performing prior open-source methods for each metric, DeepScholar-ref offers competitive Knowledge Synthesis performance, $1.09\times$ higher Relevance Rates, $2.18\times$ higher Reference Coverage, $2.08\times$ higher Citation Precision and $1.41\times$ higher Claim Coverage.

Overall, the strong relative performance of DeepScholar-ref likely reflects the efficiency of the data-processing semantic operators [37] that DeepScholar-ref uses to perform LLM-based filtering, ranking and summarization of sources to generate its report. Notably, DeepScholar-ref still demonstrates significant room for improvement and far from saturates DeepScholar-Bench, especially on key Knowledge Synthesis and Retrieval Quality metrics.

5.2 Understanding Opportunities for Improvement.

To analyze performance and opportunities for improvement, we conduct an ablation study, testing different retrievers, including two oracle settings. Table 3 shows the performance of DeepScholar-ref (GPT-4.1, Claude) and DeepScholar-ref (Llama-4), each with 3 different retrieval APIs: arxiv.org, the default in our main results, parallel.ai and tavily.com. In addition, our two oracle retrievers include the Oracle Retrieval (arxiv.org) setting and the Oracle Retrieval (All) setting, which provide the system with ArXiv references and all references, respectively, from the set of important references cited in exemplars, following our methodology for evaluating Reference Coverage.

Overall, the results demonstrate that key opportunities for improvement lie in both retrieval and knowledge synthesis capabilities. First, we see that DeepScholar-ref (GPT-4.1, Claude) with either oracle retriever nearly saturates performance on Retrieval Quality and Verifiability metrics, whereas the same method, using the arxiv.org, parallel.ai or taviliy.com retrievers, score much lower. Specifically, significant performance gaps exist for Reference Coverage and Document Importance, demonstrating the system struggles to navigate the live web and recover a diverse set of key, notable sources. Additionally, we also see that oracle retrievers improve the Nugget Coverage of either DeepScholar-ref methods by up to $1.62\times$ compared to the arxiv.org, parallel.ai or tavily.ai retrievers. Yet, the oracle settings still far from saturate Nugget Coverage, highlighting the AI system still struggle to effectively surface important facts and insights, even with high-quality sources.

5.3 Human Evaluation

To assess whether LLM-based metrics reflect human expert judgments, we conduct a human evaluation with 11 annotators, all Computer Science PhD students from four research universities across North America. In total, we collect over 300 human annotations in order to validate the LLM-based judges introduced by our automated evaluation for assessing knowledge synthesis and retrieval quality. We provide further details of our setup in Appendix A.3.4.

Agreement analysis.

Table 4 shows the results of human evaluation study as confusion matrices taken between the majority vote of human annotators and the LLM-judge. Overall, the results show the robustness of our LLM-judges based on their strong alignment with the expert human annotators. Specifically, we observe a $71.43\%$ agreement score for pairwise comparisons judging Organization, a $83.33\%$ agreement score for nugget labeling to compute Nugget Coverage, and a $65.9\%$ agreement score for labeling reference importance to compute Reference Coverage. Notably, each of these tasks require reasoning about complex academic literature and lengthy candidate related works sections. The observed agreement rates from our study provide promising results, validating the use of automated LLM-judges and metrics to assess complex generative research synthesis tasks.

For Organization, we see from the confusion matrix that the main point of human- and LLM-judges most often agree on pairwise comparisons, with strong disagreements (i.e., humans preferring the reference report and the LLM preferring the generated report or vice versa) are rare. Moreover, of the disagreements that occur between human- and LLM-judges, the LLM mis-judgments are relatively equally balanced between picking the Reference report and the Generated Report.

For Nugget Importance, we observe that in addition to the strong agreement rate observed from the confusion matrix, we also see that the human majority vote find all LLM-generated nuggets to be relevant, indicating hallucinations are rare. Moreover we see that the false postive and false negative rate of the LLM-judge are similar, and both rather small, less than $10\%$, once again indicating that severe LLM mis-labeling is rather rare.

Finally, for Reference Importance we observe an overall agreement score of $65.9\%$, and importantly false negative rate of $9.8\%$, i.e., when the LLM incorrectly labels a reference as important. The low false negative rate indicates a low likelihood of our Reference Coverage metric falsely penalizing systems. The larger off-diagonal mass (24.2%) reflects under-labeling of essential references by the LLM, suggesting that our Reference Coverage scores provide a rather conservative metric, measuring "recall" of only a subset of all truly important references for each query.

6 Related Work

Long-form Synthesis Benchmarks. While our work proposes a continually-updated, live benchmark using an automated data pipeline, several prior works instead provide expert-curated datasets for long-form research synthesis tasks, including ScholarQABench [6], OpenResearcher [66], DeepConsult [62], ResearcherBench [60], DeepResearch Bench [8], Deep Research Bench [10], SurGE [45], and LiveDRBench [16]. Unfortunately, these expert-curated benchmarks, are expensive to construct and update, can quickly become outdated, as new information becomes available, and risk data contamination, as new models are trained on publicly available data.

Alternatively, several recent benchmarks, including AcademicEval [63], LongBench-Cite [64] and SciIG [12], evaluate long-form generation tasks that do not require search over the live web, which is a key component of generative research synthesis and our benchmark. Other benchmarks focus on other long-form generation tasks, such as Wikipedia-like article generation [43], which differs substantially from our focus on complex research synthesis tasks. Crucially, unlike each of these prior works, our work proposes a live, continually-updated benchmark for evaluating generative research synthesis.

Factuality and Question Answering Benchmarks. While this work proposes a framework for studying complex, long-form research synthesis tasks, which lack an absolute notion of correctness and admit many possible reasonable answers, several recent works focus their evaluation on question answering (QA) and factuality benchmarks with short-form, easily verifiable answers. These prior benchmarks include SimpleQA [55], FRAMES [20], GAIA [31], BrowserComp [56], BrowserComp-Plus [7] WebWalkerQA [58], DeepResearch Arena [54] and others traditionally used to evaluate retrieval-augmented generation [53, 18, 61, 19, 21, 15, 49, 23]. Additionally, several benchmark develop automated dataset curation pipelines for live benchmark; however, their task focuses on short-form question-answering, as opposed to long-form report generation [36, 29, 17].

7 Conclusion

In this work, we introduced DeepScholar-bench, a live dataset and holistic, automated evaluation framework designed to rigorously benchmark an emerging class of systems designed for generative research synthesis. By automatically sourcing queries from high-quality, recent ArXiv papers, our benchmark mitigates the risks of data staleness and training contamination, while offering a real research synthesis task. Moreover, DeepScholar-bench provides an automated evaluation to holistically measure three critical dimensions: retrieval quality, knowledge synthesis and verifiability. We further release DeepScholar-ref, a reference pipeline, which we find provides a strong baseline for generative research synthesis. Overall our systematic evaluation of prior open-source systems, search agents, OpenAI’s DeepResearch and DeepScholar-ref demonstrates significant opportunities for future work, with no system surpassing a geometric mean of $31\%$ across all metrics. These results demonstrate both the difficulty of DeepScholar-bench and the exciting opportunity for further advancement in this space. We hope that DeepScholar-bench and DeepScholar-ref will support the development of more capable AI systems for generative research synthesis.

Acknowledgments

This research was supported in part by affiliate members and other supporters of the Stanford DAWN project, including Meta, Google, and VMware, as well as Cisco, SAP, and a Sloan Fellowship. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.

Appendix A Appendix

A.1 DeepScholar-bench Dataset

We provide a detailed overview and schema of the DeepScholar-bench dataset in Figure 3. In addition, Table 5 includes the papers that are included in DeepScholar-June-2025.

A.2 Overview of Baselines and Experimental Setup

We briefly overview all of the baseline systems we evaluate, which includes recent state-of-the-art generative research systems as well as DeepScholar-ref on DeepScholar-bench. We benchmark open-source research systems, including DeepResearcher [65], STORM [43] and OpenScholar [6], search agents, with Llama-4-Scout-17B-16E-Instruct [30], GPT-4.1-2025-04-14 [34], o3-2025-04-16 [34], Claude-opus-4-20250514 [3], and Gemini-2.5-pro [14] models, OpenAI’s o3-deep-research [34], and DeepScholar-ref.

We report results using GPT-4.1-2025-04-14 [34] as the judge for Nugget Coverage, and a GPT-4o-2024-08-06 [34] judge for Organization, Relevance Rate, Reference Coverage, Citation Precision and Claim Coverage. We report the Organization score as a win rate including ties, we report the strict all score for Nugget Coverage, and we report Claim Coverage with a window size of $w=1$. For all Retrieval Quality metrics, we consider the retrieved set of each given report as the set of any valid ArXiv links found within the report. To measure Document Importance, we use the OpenAlex [35] API to recover citation information. For each metric, we report an average over all reports.

A.2.1 Open-source Research Systems

We evaluate three state-of-the-art open-source systems, DeepResearcher [65], STORM [43] and OpenScholar [6]. For each, we run these systems using the Llama-4-Scout-17B-16E-Instruct model [30], which we serve with 4 A100 GPUs using vLLM [22].

DeepResearcher [65] leverages trained agents to navigate, browse and synthesize information from the web. To train an agent, this work uses end-to-end reinforcement learning and trains Qwen2.5-7B-Instruct [40]. In our benchmarks, we evaluate DeepResearcher using both the released, trained model from the authors, and using Llama-4-Scout-17B-16E-Instruct model [30] as the core LLM. We report the better performing baseline of these two, which we find in our experiments to be the Llama-4-Scout-17B-16E-Instruct backbone.

STORM [43] studies the problem of how to apply LLMs to write grounded, organized long-form articles (e.g., Wikipedia articles) from scratch. The system involves a pre-writing stage that discovers diverse research perspectives on a topic by stimulating conversations between multiple agents and leveraging web documents.

OpenScholar [6] builds a specialized retrieval-augmented LLM system for literature synthesis and scientific queries. This method includes a trained retriever from the pre-indexed peS2o [44] corpus, consisting of 45 million open-access academic papers up until October 2024, as an initial retrieval source before using web search. In our experiments, we benchmark the system using this pre-indexed corpus and limit web search to the ArXiv API.

A.2.2 Search Agents

We evaluate the following models: Llama-4-Scout-17B-16E-Instruct [30], GPT-4.1-2025-04-14 [34], o3-2025-04-16 [34], Claude-opus-4-20250514 [3], and Gemini-2.5-pro [14]. We augment each with search capabilities to ArXiv [5], and use the popular ODS framework [1] to allow the LLM to make tool calls to the search API.

A.2.3 Commercial Systems.

We focus our evaluation of commercial generative research synthesis systems on OpenAI’s o3-deep-research [34], which provides a public API allowing for our evaluation.

A.2.4 DeepScholar-ref

Similar to our evaluation of search agent we evaluate DeepScholar-ref with the following models:

Llama-4-Scout-17B-16E-Instruct [30], GPT-4.1-2025-04-14 [34], o3-2025-04-16 [34], Claude-opus-4-20250514 [3], and Gemini-2.5-pro [14]. For each of these baselines, we also use the same or a weaker model, either Llama-4 or GPT-4.1, to perform semantic filtering and top-k operators. We limit the method to two rounds of search, each with at most 2 queries.

A.3 Additional Experimental Results

A.3.1 Metadata Statistics

We provide metadata statistics characterizing the generated reports of each benchmarked method in Table 6, as well as statistics related to our evaluation metrics in Table 7.

A.3.2 Results on DeepScholar-Nov-2025

To study how our benchmark behaves under both domain and temporal shifts in the live arXiv API, we instantiate a second benchmark slice, in addition to DeepScholar-June-2025 (Section 2), which is built from 63 Computer Science papers on arXiv to study domain coverage and robustness of DeepScholar-bench in our main results.

DeepScholar-Nov-2025 contains 200 queries sampled from more than 75 distinct arXiv subject areas, spanning Computer Science, Physics, Quantitative Biology, Economics, and Quantitative Finance. We evaluate a subset of high-performing systems to check whether our conclusions remain stable: DeepScholar-ref instantiated with Llama-4-Scout-17B-16E and with GPT-4.1+o3, and the Search Agent baseline instantiated with the same model configurations.

Table 8 reports the resulting scores across all seven metrics. Overall, we observe patterns that are consistent with our main results on DeepScholar-June-2025 (Table 2). For both DeepScholar-ref and the Search Agent baseline, the o3-based variants substantially outperform their Llama-4-Scout counterparts on Organization, Nugget Coverage, Relevance Rate, Citation Precision, and Claim Coverage, while Reference Coverage and Document Importance remain broadly low across baselines. These trends mirror the relative ranking and qualitative gaps seen in our main benchmark slice, suggesting that our conclusions from DeepScholar-June-2025 generalize beyond queries related to Computer Science.

We emphasize that DeepScholar-Bench is defined by an automated data-curation and evaluation pipeline rather than a single fixed dataset. Instantiating new slices such as DeepScholar-Nov-2025 only requires specifying a set of query papers and a date range; the pipeline then automatically constructs the corresponding benchmark and produces scores under the same evaluation protocol. This design allows practitioners to easily create additional domain- or time-specific evaluations while remaining comparable to our core results.

A.3.3 Ablation Study: Understanding the Performance Impact of the Chosen Query

Our main experiments use the paper abstract as the query description $d$ (Section 3.2). A natural question is whether our conclusions depend on this particular choice of query formulation. To assess this, we run an ablation in which we replace the abstract with two alternative, realistic user queries: (i) a two-sentence summary of the paper’s key idea (Key Idea) and (ii) a single research question describing the paper’s main goal (RQ). For each paper, we prompt an LLM to convert the abstract into these two alternative query formulations. We then re-run the full benchmark for each query version and compute system-level scores for all metrics across our main baselines.

Table 9 reports Pearson correlations between system-level scores obtained under different query formulations (rows) for each metric (columns) with statistical significance testing based on a permutation-based paired test ($p{<}0.05$). Overall, we observe very strong agreement across query types: correlations are typically above $0.95$ for Organization, Nugget Coverage, Reference Coverage, Document Importance, and Claim Coverage, and above $0.77$ for Coverage Relevance Rate in all cases. All correlations between Key Idea and RQ are statistically significant, indicating that more natural, user-facing query formulations yield highly consistent system rankings.

The main sensitivity to query formulation arises for Document Importance and, to a lesser extent, Citation Precision. The abstract vs. Key Idea and abstract vs. RQ correlations on Document Importance, as well as the abstract vs. Key Idea correlation on Citation Precision, are not statistically significant, despite having large magnitudes (e.g., $r{=}0.992$ for Document Importance). This suggests that these two metrics are most sensitive to how the query is phrased, likely because small changes in the query can shift which highly-cited papers are retrieved and cited. In contrast, the remaining metrics exhibit high and statistically significant correlations across all query pairs. Taken together, these results indicate that our benchmark conclusions are broadly robust to reasonable variations in query formulation, with only modest sensitivity in how document-level importance and citation precision are expressed across different query types.

A.3.4 Human Evaluation Details

To assess whether LLM-based metrics reflect human expert judgments, we conduct a human evaluation with 11 annotators, all Computer Science PhD students from four research universities across North America. In total, we collect over 300 human annotations aimed to assess the agreement between humans and LLMs in order to validate the LLM-based judges introduced by our automated evaluation for assessing knowledge synthesis and retrieval quality. We describe our setup in detail below.

Knowledge Synthesis.

For Organization & Coherency, we sample queries and show annotators the human-written related work section alongside a system-generated report. For each pair, annotators indicate whether they prefer the system report, prefer the human-written exemplar, or consider them similarly organized. These labels are used to evaluate the pairwise comparison outcomes underlying our Organization metric. For Nugget Coverage, we first generate information nuggets from the human-written related work section. Annotators are then shown individual nuggets and asked to judge whether each nugget is vital for understanding the paper, okay, or irrelevant. These nugget-importance labels determine which nuggets are treated as essential when computing nugget coverage scores.

Reference coverage and essential citations.

To ground our Reference Coverage metric in human judgments, we ask each annotator to select a high quality paper whose related work section they are comfortable with. For that paper, the annotator identifies at least six references they consider important (i.e., references that should appear in a good related work section) and at least six references they consider not important (i.e., references that could be omitted or substituted without harming the quality of the section). These labels form gold sets of important versus non-important references, which we use both to evaluate Reference Coverage and to validate our LLM-based importance labels. We then compare human majority labels to predictions from our LLM-judge over more than 130 blind reference-level annotations. The resulting confusion matrix (rows = human labels, columns = LLM predictions) is shown in Table 4.

A.3.5 Manual Validation of LLM-based Evaluation

We study the alignment between our LLM-based evaluation and human judgments to assess the effectiveness of our automated metrics. Overall, we find that each of the metrics we introduce for the DeepScholar-bench task exhibit high agreement between LLM-based judgments and human annotations. We collect over 400 human annotations, and Table 10 shows the agreement score between human and LLMs for each LLM classification task associated with each automated metrics. The results demonstrate above $70\%$ agreement scores across each. We additionally compute the nugget precision and grounded-ness scores, following prior work [46], observing scores of $.83$ and $1.0$ respectively, demonstrating that the generated nuggets are accurate and do not contain hallucinations.

A.3.6 Agreement Between Different LLM Judges

To test the robustness of our evaluation to the choice of LLM judge, we repeat all experiments with three different judges: GPT-4o, Llama-4-17b-16e-Instruct, and DeepSeek-R1-Distill-Qwen-32B. For each pair of judges, we compute the Pearson correlation between system-level scores across five baselines (DeepScholar-ref (GPT-4.1, Claude), DeepScholar-ref (Llama-4), Search Agent (Claude), Search Agent (Llama-4), and OpenAI DeepResearch for all metrics except document importance, which is not LLM-based. We then run perturbation-based permutation tests to assess whether these correlations are significantly greater than zero. The resulting correlations and significance markers are shown in Table 11.

Correlations are generally very high (often above 0.9). Notably GPT-4o vs. DeepSeek shows statistically significant correlations across all the metrics at the $p<0.05$ level, indicating that these two judges high agreement. Most metrics also remain significantly correlated, but we observe the main sensitivity to the choice of judge on nugget coverage (non-significant for both GPT-4o vs. Llama-4-17b-16e-Instruct and DeepSeek-R1-Distill-Qwen-32B vs. Llama-4-17b-16e-Instruct). In addition, relevance rate shows non-statistically significant correlation for GPT-4o vs. Llama-4-17b-16e-Instruct and similarly Citation Precision for DeepSeek-R1-Distill-Qwen-32B vs. Llama-4-17b-16e-Instruct. It is worth mentioning that even in these cases, the correlations remain large (all above 0.41), suggesting that discrepancies are not severe.

A.4 Evaluation Details

Here we provide the detailed implementation and prompts for the evaluation metrics used in DeepScholar-Bench. In the following, we also run some ablation studies on different metrics.

• •

Knowledge Synthesis – Organization: Box A.4.3 shows the prompt used to compare system-generated reports with the human-written exemplar, yielding the win-rate of each system based on organization and coherence.

• •

Knowledge Synthesis – Nugget Coverage: For this metric we follow the nugget-based evaluation prompts from prior work [39], which extract essential facts from the exemplar and check their presence in the generated report.

• •

Retrieval Quality – Document Importance: We select important references using the LOTUS program shown in Figure 5.

• •

Retrieval Quality – Relevance Rate: Following the Cranfield model of relevance, in Box A.4.3 we prompt the model for assigning graded relevance scores (0–2) to each retrieved source.

• •

Verifiability – Citation Precision and Claim Coverage: Following prior work [11, 26], we check entailment of claims with respect to cited sources. using the prompt shown in Box A.4.3 for both Citation Precision and Claim Coverage as explained in Section 3.3.

We note that Reference Coverage and Document Importance are computed deterministically as explained in Section 3.2: the former by comparing the retrieved set against the important references in the human-written exemplar, and the latter by normalizing the number of citations of the reference papers.

A.4.1 Reference Coverage

Figure 4 illustrates the distribution of important citations across the human exemplar reports in DeepScholar-Bench. For each exemplar, we used the LOTUS program shown in Figure 5 to identify which citations are important and therefore essential to include in a high-quality related work section. We then separate these important citations into two groups: those that appear on ArXiv (shown in red) and those that do not (shown in orange). The blue portion of each bar corresponds to non-essential citations, as determined by the same Lotus-based procedure.

The plot highlights two consistent trends. First, many exemplar related work sections contain a large number of non-essential citations. While such references may be useful for narrative flow or broader context, they are not indispensable. Non-essential citations can be somewhat subjective, depending on how authors choose to frame the story of their paper. In contrast, the important citations represent the “must-have” references i.e., the foundational works in the field that are necessary for situating the contribution. Second, we observe that the red segments (important ArXiv citations) are well distributed across exemplars, indicating that ArXiv is a reliable and sufficiently broad source for recovering many of the essential references.

A.4.2 Ablation Study on Verifiability

In the main paper (Section 3.3, we reported verifiability metrics results using a sliding window of size $w=1$ when computing claim coverage. That is, for each claim sentence, we considered a citation to be valid if any of the references in the same sentence or within one sentence before or after sufficiently supported the claim. Here, we extend this analysis to study the effect of varying the window size. Specifically, we report the citation coverage achieved by different systems when the window size ranges from $w=0$ (same-sentence only) up to $w=5$ (five sentences before or after).

As shown in Figure 6, increasing the window size consistently improves citation coverage across all baselines. This is expected: the larger the window, the higher the probability that one of the cited references in the $[-w,+w]$ neighborhood of a claim provides sufficient support. However, we also note that very large window sizes are less desirable in practice, as they often correspond to references being far from the claims they are intended to support, reducing readability and making it harder for readers to verify the connection between claims and citations. Moreover, from Table 6, we see that real academic writing tends to be densely cited, with at least one citation on average per sentence in the human exemplars. Overall, the results of our ablation study highlight the trade-off between stricter precision ($w=0$) and more lenient recall-oriented settings ($w\geq 1$).

A.4.3 Document Importance Across Human Exemplars

In this section, we illustrate the distribution of document importance, measured by the number of citations of references in the human-written exemplars in DeepScholar-Bench. Figure 7 reports two histograms: (a) the distribution of citation counts across all references, and (b) the distribution restricted to references that appear on ArXiv. We plot the logarithm of citation counts, with values obtained from the OpenAlex API [35], an open and widely used scholarly database that provides citation-level metadata. While citation counts in OpenAlex may not exactly match those from other sources such as Google Scholar, the relative counts are consistent, making it a reliable open-source alternative.

As shown in Figure 7, the distribution is highly skewed due to a small number of papers with exceptionally large citation counts (e.g., over 10k citations). These outliers inflate the mean citation values, resulting in relatively high averages compared to typical references (478.3 citations across all references and 647.6 for ArXiv-only references). In contrast, the median values are lower (31 for all references and 36 for ArXiv-only). This skew highlights the challenge of using citation counts as a proxy for importance, as the median citation count of references, among different human-written exemplars exhibits high variance.

A.5 Extended Description of Baselines

We provide an extended description of each benchmarked method, including relevant implementation details, and parameters used in our evaluation.

A.5.1 DeepResearcher

The DeepResearcher pipeline follows a structured tool-augmented reasoning framework designed for iterative web-based information retrieval. The system mandates explicit reasoning before any tool invocation, with reasoning encapsulated in <think> tags to ensure interpretability and control. After reasoning, the model generates a JSON-formatted request specifying the “web search” tool and its query. These queries are executed via the Lotus Search API, which we replaced with an ArXiv-specific search interface to provide a controlled retrieval API for our evaluation. Retrieved results are returned in a structured format containing the title, URL, and snippet, and are stored in memory for reference across subsequent reasoning steps. This iterative process continues until the model determines that sufficient evidence has been gathered, after which a synthesized final response is produced.

For our experiments, we used Llama-4-Scout-17B-16E-Instruct as the base model, replacing the originally proposed DeepResearcher-7b, since it demonstrated consistently better retrieval-augmented reasoning performance in our experiments. The prompt was slightly modified to align with LLama-4 prompt style as detailed in Box  A.5.1. The retrieval depth was set to 10 sources per query, which is the default in the system and provides a balanced trade-off between coverage and efficiency. We restricted each query to a single rollout with a maximum of 10 steps, following the DeepResearcher defaults; this limit is generous as most rollouts converge in fewer than three steps, but it ensures the system has headroom for more complex queries. The default web search API was replaced with ArXiv search to comply with our benchmark settings.

A.5.2 Openscholar

The OpenScholar pipeline follows a four-stage process: initial retrieval, response and feedback generation, iterative refinement, and citation verification. In the first stage, text segments are retrieved from a fixed index using a contriever model, which encodes texts and retrieves passages based on semantic similarity. These passages are reranked and used to generate an initial draft response, where citations are aligned with the supporting passages. The second stage introduces feedback generation, where the model produces up to three feedback statements highlighting potential improvements in the draft, such as missing content or organization issues; if additional evidence is required, retrieval queries are issued. The third stage iteratively refines the response by conditioning on the previous draft, retrieved passages, and newly added evidence, yielding improved responses at each step until feedback has been fully incorporated. Finally, citation verification ensures that all citation-worthy statements are adequately grounded in the retrieved sources, inserting additional citations where necessary without removing content.

For consistency with other baselines, we employ the Llama-4-Scout-17B-16E-Instruct model for generation. The retrieval pipeline initially collects 100 text segments from peS2o_v3 using the default pes2o_contriever111https://huggingface.co/akariasai/pes2o_contriever model. The reranker used is OpenScholar_Reranker222https://huggingface.co/OpenSciLM/OpenScholar_Reranker, also kept at its default setting. To align parameterization across baselines, we increase the number of sources used in generation (top_n) from 10 to 30. Furthermore, the default search API is replaced with the arXiv API, to provide a controlled retrieval corpus and API in our experiments.

A.5.3 Search Agent

Search Agents are implemented using each LLM API with search API access. We implement a ReAct Agent, instantiated from the smolagents.ToolCallingAgent, with a system prompt based on the open-source OpenDeepSearch (ODS) framework designed for deep web search and retrieval, along with the search agent as an external tool. At each reasoning step, the ReAct agent can either invoke the search agent through a web_search action or decide to produce a final_answer. The search agent interfaces with the search API to fetch relevant academic articles given a query, after which an LLM generates concise summaries of the retrieved content. To maintain consistency with the benchmark setting, the standard search API was replaced with the arXiv API. The regular search agent fails when tasked with full abstract queries; hence the ReAct-based agent was employed, which generates shorter, more effective searchable queries. The agent keeps track of retrieved results across turns, allowing references to past evidence during the reasoning process. After a maximum of 5 iterations, the agent is compelled to conclude with a final response, ensuring bounded computational steps.

For the parameterization of the Search Agents, we set the search agent to retrieve 30 results per query, which is more generous than the default in order to establish fair comparability with other baselines. The maximum iteration limit was fixed at 5, consistent with the default setup of the ODS framework, providing sufficient exploration without excessive search depth. The ReAct prompt was slightly modified to tailor to the specific use of the ArXiv search API, as presented in Box  A.5.3, A.5.3 and A.5.3.

A.5.4 STORM

The STORM pipeline follows a structured multi-stage process to generate comprehensive, Wikipedia-style articles from a given topic. First, related Wikipedia articles are retrieved and their TOCs are clustered to identify candidate perspectives, which act as anchors for exploration. This is followed by Simulated Multi-turn Conversations where an LLM plays both the question-asking and answering roles, querying a retrieval module and synthesizing evidence-based responses. Parallel to this, the model generates a draft outline purely from its parametric knowledge in the Draft Outline Generation stage. The outline is then refined by grounding it with retrieved evidence and conversation outputs. In the final step, each section is drafted with explicit inline citations drawing on both parametric knowledge and retrieved references. All the sections are concatenated together to form the final result.

For parameter settings, we used STORM’s default configurations wherever possible to preserve fidelity to its design: a maximum of 3 turns per perspective, 3 perspectives, and up to 3 search queries per turn. For search, we considered the top 15 results for each query, ensuring a reasonable breadth without overwhelming the pipeline. To make STORM comparable with other baselines, we raised the number of collected references per section title to 30 (more generous than the default), as this allows for richer evidence integration during drafting. Importantly, we replaced the original search API with arXiv search to control the retrieval API for our benchmark settings. Finally, we use Llama-4-Scout-17B-16E-Instruct as the base model.

A.5.5 OpenAI’s DeepResearch

We use OpenAI’s DeepResearch system based on the o3-deep-research333https://platform.openai.com/docs/models/o3-deep-research model with a custom MCP to only search ArXiv and return $n=30$ results per query. To prevent the model from getting search results after the given paper was uploaded, the MCP used a custom endpoint to set the latest date that it should retrieve. All other settings were set to default values.

A.6 DeepScholar-ref Details and Configurations

DeepScholar-base operates through three main stages: retrieval, filtering, and final generation (Figure 2).

Retrieval

In this stage, an LLM generates $Q$ search queries conditioned on the input abstract and summaries of prior retrievals. Each query is submitted to the configured search API (ArXiv, tavily, etc.) to obtain up to $search\_K$ relevant papers within the specified date range. The code and prompt used for this step are provided in Figure 8 and Box A.7 respectively. This process is repeated $N$ times.

Filtering

Retrieved results are refined using two semantic operators from LOTUS [37, 28]: Sem-Filter and Sem-TopK, which together select the top $K$ most relevant papers. The code is given in Figure 9.

Final Generation

The filtered set of papers is then aggregated via a Sem-Agg query to produce the final output. The corresponding code for this step is shown in Figure 10 with prompt in Box  A.7.

Unless otherwise specified, the pipeline parameters are set to $Q=2$, $search\_K=50$, $N=2$, and $K=30$.

A.7 Examples of Generated Reports

We provide examples of generated reports from different systems for paper 0 in our dataset (according to Table 5 i.e., ‘TaxAgent: How Large Language Model Designs Fiscal Policy’ with Arxiv ID of 2506.02838. Figures LABEL:fig:example:ours, 11, 12, 13 and 14 display the generated reports produced by DeepScholar-ref, Search Agent, DeepResearcher, OpenScholar, and Storm respectively, all using Llama-4.