FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering

TL;DR

FinAgentBench is a large-scale benchmark dataset designed to evaluate multi-step, agentic retrieval capabilities of language models in the financial domain, addressing a critical gap in domain-specific information retrieval evaluation.

Contribution

It introduces the first benchmark for agentic retrieval in finance, with a dataset of 26K annotated examples, and evaluates models' ability to identify relevant documents and key passages.

Findings

State-of-the-art models show room for improvement in agentic retrieval.

Targeted fine-tuning significantly enhances retrieval performance.

Benchmark enables detailed analysis of LLM behavior in finance retrieval tasks.

Abstract

Accurate information retrieval (IR) is critical in the financial domain, where investors must identify relevant information from large collections of documents. Traditional IR methods -- whether sparse or dense -- often fall short in retrieval accuracy, as it requires not only capturing semantic similarity but also performing fine-grained reasoning over document structure and domain-specific knowledge. Recent advances in large language models (LLMs) have opened up new opportunities for retrieval with multi-step reasoning, where the model ranks passages through iterative reasoning about which information is most relevant to a given query. However, there exists no benchmark to evaluate such capabilities in the financial domain. To address this gap, we introduce FinAgentBench, the first large-scale benchmark for evaluating retrieval with multi-step reasoning in finance -- a setting we term agentic retrieval. The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms and assesses whether LLM agents can (1) identify the most relevant document type among candidates, and (2) pinpoint the key passage within the selected document. Our evaluation framework explicitly separates these two reasoning steps to address context limitations. This design enables to provide a quantitative basis for understanding retrieval-centric LLM behavior in finance. We evaluate a suite of state-of-the-art models and further demonstrated how targeted fine-tuning can significantly improve agentic retrieval performance. Our benchmark provides a foundation for studying retrieval-centric LLM behavior in complex, domain-specific tasks for finance.