Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents

TL;DR

This paper introduces TRQA, a comprehensive evaluation framework and benchmark for deep research agents that perform multi-step reasoning over large sources, addressing current evaluation challenges with structured, large-scale datasets.

Contribution

It proposes a new evaluation framework satisfying key requirements and creates TRQA, a benchmark combining real-world and synthetic data for assessing deep research agents.

Findings

Established baseline retrieval and end-to-end results.

Demonstrated the framework's ability to evaluate complex research agents.

Provided a scalable, structured approach for future evaluations.

Abstract

Deep research agents have emerged as LLM-based systems designed to perform multi-step information seeking and reasoning over large, open-domain sources to answer complex questions by synthesizing information from multiple information sources. Given the complexity of the task and despite various recent efforts, evaluation of deep research agents remains fundamentally challenging. This paper identifies a list of requirements and optional properties for evaluating deep research agents. We observe that existing benchmarks do not satisfy all identified requirements. Inspired by prior research on TREC Total Recall Tracks, we introduce the task of Total Recall Question Answering and develop a framework for deep research agents evaluation that satisfies the identified criteria. Our framework constructs single-answer, total recall queries with precise evaluation and relevance judgments derived from a structured knowledge base paired with a text corpus, enabling large-scale data construction. Using this framework, we build TRQA, a deep research benchmark constructed from Wikidata-Wikipedia as a real-world source and a synthetically generated e-commerce knowledge base and corpus to mitigate the effects of data contamination. We benchmark the collection with representative retriever and deep research models and establish baseline retrieval and end-to-end results for future comparative evaluation.