FinTextQA: A Dataset for Long-form Financial Question Answering

TL;DR

FinTextQA introduces a new comprehensive dataset for long-form financial question answering and evaluates retrieval-augmented generation systems, highlighting their performance and robustness under noisy conditions.

Contribution

This paper presents FinTextQA, a novel dataset for financial LFQA, and benchmarks a RAG-based system, demonstrating its effectiveness and resilience to noise.

Findings

Baichuan2-7B performs comparably to GPT-3.5-turbo in accuracy.

Optimal system configuration includes Ada2, Automated Merged Retrieval, Bge-Reranker-Base, and Baichuan2-7B.

Models become less affected by noise when context length exceeds a certain threshold.

Abstract

Accurate evaluation of financial question answering (QA) systems necessitates a comprehensive dataset encompassing diverse question types and contexts. However, current financial QA datasets lack scope diversity and question complexity. This work introduces FinTextQA, a novel dataset for long-form question answering (LFQA) in finance. FinTextQA comprises 1,262 high-quality, source-attributed QA pairs extracted and selected from finance textbooks and government agency websites.Moreover, we developed a Retrieval-Augmented Generation (RAG)-based LFQA system, comprising an embedder, retriever, reranker, and generator. A multi-faceted evaluation approach, including human ranking, automatic metrics, and GPT-4 scoring, was employed to benchmark the performance of different LFQA system configurations under heightened noisy conditions. The results indicate that: (1) Among all compared generators, Baichuan2-7B competes closely with GPT-3.5-turbo in accuracy score; (2) The most effective system configuration on our dataset involved setting the embedder, retriever, reranker, and generator as Ada2, Automated Merged Retrieval, Bge-Reranker-Base, and Baichuan2-7B, respectively; (3) models are less susceptible to noise after the length of contexts reaching a specific threshold.