FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents

TL;DR

FinResearchBench introduces a logic tree-based evaluation framework specifically designed for financial research agents, enabling comprehensive, automatic assessment across diverse complex financial research tasks, filling a critical gap in current evaluation methods.

Contribution

It presents the first Agent-as-a-Judge system using logic trees for financial research evaluation, covering 70 questions across 7 task types for systematic assessment.

Findings

Provides a reliable and robust evaluation method.

Automatically assesses research agents across multiple financial tasks.

Enhances understanding of agent capabilities in finance.

Abstract

Recently, AI agents are rapidly evolving in intelligence and widely used in professional research applications, such as STEM, software development, and finance. Among these AI agents, deep research agent is a key category as it can perform long-horizon tasks and solve problems of greater complexity. However, there are few evaluation frameworks and benchmarks that systematically and automatically investigate the capabilities of these research agents. In addition, financial research problems have distinct complexity and subtlety. To fill in the gap, we propose FinResearchBench, which is a logic tree-based Agent-as-a-Judge and targets specifically for the financial research agents. It provides a comprehensive and automatic assessment of the research agents across 7 key types of tasks in the financial research domain. The contributions of this work are two-folded: (1) the first and innovative Agent-as-a-Judge system that extracts the logic tree of the research outcome and uses it as the intermediate information to present a comprehensive, reliable, and robust evaluation; (2) finance-oriented that it covers 70 typical financial research questions, spreading across 7 frequently encountered types of task in the domain.