Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering

TL;DR

This paper investigates retrieval failures in financial long-document question answering, analyzing different retrieval granularities, introducing an oracle analysis, and proposing a domain-specific page scorer to improve retrieval accuracy.

Contribution

It systematically studies within-document retrieval failures, evaluates multiple retrieval strategies, and introduces a fine-tuned page scorer to enhance retrieval performance in financial QA.

Findings

Improved page and chunk retrieval with the new page scorer.

Oracle analysis reveals room for better page and chunk retrieval.

Diverse retrieval strategies show gains in document discovery and recall.

Abstract

Retrieval-augmented generation is increasingly used for financial question answering over long regulatory filings, yet reliability depends on retrieving the exact context needed to justify answers in high stakes settings. We study a frequent failure mode in which the correct document is retrieved but the page or chunk that contains the answer is missed, leading the generator to extrapolate from incomplete context. Despite its practical significance, this within-document retrieval failure mode has received limited systematic attention in the Financial Question Answering (QA) literature. We evaluate retrieval at multiple levels of granularity, document, page, and chunk level, and introduce an oracle based analysis to provide empirical upper bounds on retrieval and generative performance. On a 150 question subset of FinanceBench, we reproduce and compare diverse retrieval strategies including dense, sparse, hybrid, and hierarchical methods with reranking and query reformulation. Across methods, gains in document discovery tend to translate into stronger page recall, yet oracle performance still suggests headroom for page and chunk level retrieval. To target this gap, we introduce a domain fine-tuned page scorer that treats pages as an intermediate retrieval unit between documents and chunks. Unlike prior passage-based hierarchical retrieval, we fine-tune a bi-encoder specifically for page-level relevance on financial filings, exploiting the semantic coherence of pages. Overall, our results demonstrate a significant improvement in page recall and chunk retrieval.