Integrating Domain Knowledge for Financial QA: A Multi-Retriever RAG Approach with LLMs

TL;DR

This paper presents a multi-retriever RAG system combined with LLMs to improve financial numerical reasoning QA by integrating domain knowledge, achieving state-of-the-art results and surpassing previous models.

Contribution

The study introduces a multi-retriever RAG approach with domain-specific training that significantly enhances financial QA performance over existing models.

Findings

Domain-specific training with SecBERT improves neural symbolic model performance.

Prompt-based LLM achieves SOTA with >7% improvement but below human level.

External knowledge benefits outweigh hallucination risks in larger models.

Abstract

This research project addresses the errors of financial numerical reasoning Question Answering (QA) tasks due to the lack of domain knowledge in finance. Despite recent advances in Large Language Models (LLMs), financial numerical questions remain challenging because they require specific domain knowledge in finance and complex multi-step numeric reasoning. We implement a multi-retriever Retrieval Augmented Generators (RAG) system to retrieve both external domain knowledge and internal question contexts, and utilize the latest LLM to tackle these tasks. Through comprehensive ablation experiments and error analysis, we find that domain-specific training with the SecBERT encoder significantly contributes to our best neural symbolic model surpassing the FinQA paper's top model, which serves as our baseline. This suggests the potential superior performance of domain-specific training. Furthermore, our best prompt-based LLM generator achieves the state-of-the-art (SOTA) performance with significant improvement (>7%), yet it is still below the human expert performance. This study highlights the trade-off between hallucinations loss and external knowledge gains in smaller models and few-shot examples. For larger models, the gains from external facts typically outweigh the hallucination loss. Finally, our findings confirm the enhanced numerical reasoning capabilities of the latest LLM, optimized for few-shot learning.