Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks

TL;DR

The paper introduces the Finance Agent Benchmark, a comprehensive dataset and evaluation framework for testing large language models on real-world financial research tasks involving SEC filings and financial analysis.

Contribution

It presents a new benchmark with expert-validated questions and an agentic setup, highlighting current AI limitations in financial analysis tasks.

Findings

Best model achieved 46.8% accuracy

Average cost per query was $3.79

Significant room for improvement in AI financial capabilities

Abstract

Artificial Intelligence (AI) technology has emerged as a transformative force in financial analysis and the finance industry, though significant questions remain about the full capabilities of Large Language Model (LLM) agents in this domain. We present the Finance Agent Benchmark, featuring challenging and diverse real-world finance research problems that require LLMs to perform complex analysis using recent SEC filings. We construct the benchmark using a taxonomy of nine financial task categories, developed in consultation with experts from banks, hedge funds, and private equity firms. The dataset includes 537 expert-authored questions covering tasks from information retrieval to complex financial modeling, each validated through a rigorous review process to ensure accuracy and relevance. Moreover, we implement an agentic harness that equips LLMs with tools sufficient to produce accurate responses, including Google Search and EDGAR database access. Overall, the Finance Agent Benchmark provides a comprehensive testbed for measuring the progress of LLM-driven finance agents. Our evaluation reveals significant limitations in current AI capabilities - even the best-performing model (OpenAI o3) achieved only 46.8% accuracy at an average cost of $3.79 per query. This underscores the need for further advancements before reliable deployment in high-stakes finance settings.