FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline

TL;DR

FinQAPT is an end-to-end pipeline that uses advanced techniques and LLMs to improve financial report analysis and question answering, achieving state-of-the-art accuracy on a benchmark dataset.

Contribution

The paper introduces FinQAPT, a novel pipeline with clustering-based negative sampling and Dynamic N-shot Prompting, enhancing financial question answering performance.

Findings

Achieved 80.6% accuracy on FinQA dataset.

State-of-the-art accuracy at module level.

Identified challenges in context extraction affecting pipeline performance.

Abstract

Financial decision-making hinges on the analysis of relevant information embedded in the enormous volume of documents in the financial domain. To address this challenge, we developed FinQAPT, an end-to-end pipeline that streamlines the identification of relevant financial reports based on a query, extracts pertinent context, and leverages Large Language Models (LLMs) to perform downstream tasks. To evaluate the pipeline, we experimented with various techniques to optimize the performance of each module using the FinQA dataset. We introduced a novel clustering-based negative sampling technique to enhance context extraction and a novel prompting method called Dynamic N-shot Prompting to boost the numerical question-answering capabilities of LLMs. At the module level, we achieved state-of-the-art accuracy on FinQA, attaining an accuracy of 80.6%. However, at the pipeline level, we observed decreased performance due to challenges in extracting relevant context from financial reports. We conducted a detailed error analysis of each module and the end-to-end pipeline, pinpointing specific challenges that must be addressed to develop a robust solution for handling complex financial tasks.