Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents

TL;DR

Deep Research Comparator is a platform that enables detailed evaluation and comparison of deep research agents, including their intermediate steps, to improve assessment and development of such AI systems.

Contribution

The paper introduces a comprehensive platform for evaluating deep research agents with fine-grained human feedback and a baseline agent scaffold for easy integration of large language models.

Findings

Collected real user preference data from 17 annotators.

Demonstrated the platform's utility through a practical evaluation.

Provided a baseline agent scaffold for research and development.

Abstract

Effectively evaluating deep research agents that autonomously search the web, analyze information, and generate reports remains a major challenge, particularly when it comes to assessing long reports and giving detailed feedback on their intermediate steps. To address these gaps, we introduce Deep Research Comparator, a platform that offers a holistic framework for deep research agent hosting, side-by-side comparison, fine-grained human feedback collection, and ranking calculation. Given a user query, our platform displays the final reports from two different agents along with their intermediate steps during generation. Annotators can evaluate the overall quality of final reports based on side-by-side comparison, and also provide detailed feedback separately by assessing intermediate steps or specific text spans within the final report. Furthermore, we develop Simple Deepresearch, an end-to-end agent scaffold. This scaffold serves as a baseline that facilitates the easy integration of various large language models to transform them into deep research agents for evaluation. To demonstrate the platform's utility for deep research agent development, we have collected real user preference data from 17 annotators on three deep research agents. A demo video of our platform can be found at https://www.youtube.com/watch?v=g4d2dnbdseg.