FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents

TL;DR

FinDVer is a new benchmark for evaluating large language models' ability to verify claims and provide explanations in complex, long, and hybrid financial documents, highlighting current limitations and guiding future improvements.

Contribution

The paper introduces FinDVer, a comprehensive benchmark with expert-annotated data for assessing explainable claim verification in financial documents using LLMs.

Findings

GPT-4o performs below human experts in claim verification.

Long-context and RAG settings pose significant challenges for LLMs.

Analysis reveals common reasoning errors in current models.

Abstract

We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 2,400 expert-annotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning, each addressing common scenarios encountered in real-world financial contexts. We assess a broad spectrum of LLMs under long-context and RAG settings. Our results show that even the current best-performing system, GPT-4o, still lags behind human experts. We further provide in-depth analysis on long-context and RAG setting, Chain-of-Thought reasoning, and model reasoning errors, offering insights to drive future advancements. We believe that FinDVer can serve as a valuable benchmark for evaluating LLMs in claim verification over complex, expert-domain documents.