Enhancing Financial Question Answering with a Multi-Agent Reflection Framework

TL;DR

This paper introduces a multi-agent reflection framework that significantly improves financial question answering performance of LLMs by incorporating multiple critic agents for better reasoning and answer evaluation.

Contribution

The study proposes a novel multi-agent framework with multiple critic agents that reflect on reasoning steps, enhancing financial QA accuracy over single-agent systems.

Findings

Performance increased by 15% for LLaMA3-8B

Achieved parity or surpassing larger models like LLaMA3.1-405B and GPT-4o-mini

Slightly below Claude-3.5 Sonnet in some cases

Abstract

While Large Language Models (LLMs) have shown impressive capabilities in numerous Natural Language Processing (NLP) tasks, they still struggle with financial question answering (QA), particularly when numerical reasoning is required. Recently, LLM-based multi-agent frameworks have demonstrated remarkable effectiveness in multi-step reasoning, which is crucial for financial QA tasks as it involves extracting relevant information from tables and text and then performing numerical reasoning on the extracted data to infer answers. In this study, we propose a multi-agent framework incorporating a critic agent that reflects on the reasoning steps and final answers for each question. Additionally, we enhance our system by adding multiple critic agents, each focusing on a specific aspect of the answer. Our results indicate that this framework significantly improves performance compared to single-agent reasoning, with an average performance increase of 15% for the LLaMA3-8B model and 5% for the LLaMA3-70B model. Furthermore, our framework performs on par with, and in some cases surpasses, larger single-agent LLMs such as LLaMA3.1-405B and GPT-4o-mini, though it falls slightly short compared to Claude-3.5 Sonnet. Overall, our framework presents an effective solution to enhance open-source LLMs for financial QA tasks, offering a cost-effective alternative to larger models like Claude-3.5 Sonnet.