Enhancing Financial Report Question-Answering: A Retrieval-Augmented Generation System with Reranking Analysis

TL;DR

This paper develops a retrieval-augmented generation system for financial report question-answering, demonstrating that neural reranking significantly enhances answer correctness and reduces errors on the FinDER benchmark.

Contribution

It introduces a hybrid retrieval and reranking pipeline that improves financial report question-answering accuracy over baseline methods.

Findings

Reranking increases correctness from 33.5% to 49.0%.

Error rate for incorrect answers drops from 35.3% to 22.5%.

System outperforms baseline retrieval strategies.

Abstract

Financial analysts face significant challenges extracting information from lengthy 10-K reports, which often exceed 100 pages. This paper presents a Retrieval-Augmented Generation (RAG) system designed to answer questions about S&P 500 financial reports and evaluates the impact of neural reranking on system performance. Our pipeline employs hybrid search combining full-text and semantic retrieval, followed by an optional reranking stage using a cross-encoder model. We conduct systematic evaluation using the FinDER benchmark dataset, comprising 1,500 queries across five experimental groups. Results demonstrate that reranking significantly improves answer quality, achieving 49.0 percent correctness for scores of 8 or above compared to 33.5 percent without reranking, representing a 15.5 percentage point improvement. Additionally, the error rate for completely incorrect answers decreases from 35.3 percent to 22.5 percent. Our findings emphasize the critical role of reranking in financial RAG systems and demonstrate performance improvements over baseline methods through modern language models and refined retrieval strategies.