Query Generation Pipeline with Enhanced Answerability Assessment for Financial Information Retrieval

TL;DR

This paper introduces a methodology for creating domain-specific information retrieval benchmarks using LLM-based query generation, exemplified by KoBankIR, to improve financial IR evaluation and highlight retrieval challenges.

Contribution

It presents a systematic approach for constructing financial IR benchmarks via LLM-generated queries and an enhanced answerability assessment, improving alignment with human judgments.

Findings

Existing models struggle with complex multi-document queries in KoBankIR.

The proposed pipeline achieves better alignment with human judgments than prior methods.

KoBankIR contains 815 queries from 204 banking documents.

Abstract

As financial applications of large language models (LLMs) gain attention, accurate Information Retrieval (IR) remains crucial for reliable AI services. However, existing benchmarks fail to capture the complex and domain-specific information needs of real-world banking scenarios. Building domain-specific IR benchmarks is costly and constrained by legal restrictions on using real customer data. To address these challenges, we propose a systematic methodology for constructing domain-specific IR benchmarks through LLM-based query generation. As a concrete implementation of this methodology, our pipeline combines single and multi-document query generation with an enhanced and reasoning-augmented answerability assessment method, achieving stronger alignment with human judgments than prior approaches. Using this methodology, we construct KoBankIR, comprising 815 queries derived from 204 official banking documents. Our experiments show that existing retrieval models struggle with the complex multi-document queries in KoBankIR, demonstrating the value of our systematic approach for domain-specific benchmark construction and underscoring the need for improved retrieval techniques in financial domains.