Multi-Reranker: Maximizing performance of retrieval-augmented generation in the FinanceRAG challenge

TL;DR

This paper presents a high-performance, finance-specific Retrieval-Augmented Generation system optimized through ablation studies and reranker models, achieving second place in the ACM-ICAIF '24 FinanceRAG challenge.

Contribution

It introduces novel methods for query expansion, corpus refinement, and long-context management to enhance retrieval accuracy and response quality in financial data analysis.

Findings

Achieved 2nd place in the FinanceRAG Challenge

Improved retrieval accuracy with multiple rerankers

Enhanced long-context handling for better responses

Abstract

As Large Language Models (LLMs) increasingly address domain-specific problems, their application in the financial sector has expanded rapidly. Tasks that are both highly valuable and time-consuming, such as analyzing financial statements, disclosures, and related documents, are now being effectively tackled using LLMs. This paper details the development of a high-performance, finance-specific Retrieval-Augmented Generation (RAG) system for the ACM-ICAIF '24 FinanceRAG competition. We optimized performance through ablation studies on query expansion and corpus refinement during the pre-retrieval phase. To enhance retrieval accuracy, we employed multiple reranker models. Notably, we introduced an efficient method for managing long context sizes during the generation phase, significantly improving response quality without sacrificing performance. We ultimately achieve 2nd place in the FinanceRAG Challenge. Our key contributions include: (1) pre-retrieval ablation analysis, (2) an enhanced retrieval algorithm, and (3) a novel approach for long-context management. This work demonstrates the potential of LLMs in effectively processing and analyzing complex financial data to generate accurate and valuable insights. The source code and further details are available at https://github.com/cv-lee/FinanceRAG.