Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles
Shaohan Wang, Benfeng Xu, Licheng Zhang, Mingxuan Du, Chiwei Zhu, Xiaorui Wang, Zhendong Mao, Yongdong Zhang

TL;DR
The paper introduces Wiki Live Challenge, a new benchmark using expert-level Wikipedia articles to evaluate Deep Research Agents' ability to produce high-quality, verifiable research content, highlighting current gaps and advancing the field.
Contribution
It presents a novel live benchmark with expert-verified Wikipedia articles and a detailed evaluation framework for assessing research agents.
Findings
Current DRAs lag behind human expert-level articles.
The WLC benchmark effectively identifies gaps in DRA capabilities.
Extensive experiments validate the benchmark's utility.
Abstract
Deep Research Agents (DRAs) have demonstrated remarkable capabilities in autonomous information retrieval and report generation, showing great potential to assist humans in complex research tasks. Current evaluation frameworks primarily rely on LLM-generated references or LLM-derived evaluation dimensions. While these approaches offer scalability, they often lack the reliability of expert-verified content and struggle to provide objective, fine-grained assessments of critical dimensions. To bridge this gap, we introduce Wiki Live Challenge (WLC), a live benchmark that leverages the newest Wikipedia Good Articles (GAs) as expert-level references. Wikipedia's strict standards for neutrality, comprehensiveness, and verifiability serve as a great challenge for DRAs, with GAs representing the pinnacle of which. We curate a dataset of 100 recent Good Articles and propose Wiki Eval, a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Artificial Intelligence in Healthcare and Education · Biomedical Text Mining and Ontologies
