FinBERT-QA: Financial Question Answering with pre-trained BERT Language Models

TL;DR

FinBERT-QA is a financial question answering system that leverages pre-trained BERT models to improve answer relevance and retrieval accuracy in financial data, outperforming previous methods on key metrics.

Contribution

The paper introduces a novel financial QA system combining BM25 retrieval with BERT-based re-ranking, and demonstrates significant performance improvements on financial datasets.

Findings

Achieved 16% improvement in MRR

Achieved 17% improvement in NDCG

Achieved 21% improvement in Precision@1

Abstract

Motivated by the emerging demand in the financial industry for the automatic analysis of unstructured and structured data at scale, Question Answering (QA) systems can provide lucrative and competitive advantages to companies by facilitating the decision making of financial advisers. Consequently, we propose a novel financial QA system using the transformer-based pre-trained BERT language model to address the limitations of data scarcity and language specificity in the financial domain. Our system focuses on financial non-factoid answer selection, which retrieves a set of passage-level texts and selects the most relevant as the answer. To increase efficiency, we formulate the answer selection task as a re-ranking problem, in which our system consists of an Answer Retriever using BM25, a simple information retrieval approach, to first return a list of candidate answers, and an Answer Re-ranker built with variants of pre-trained BERT language models to re-rank and select the most relevant answers. We investigate various learning, further pre-training, and fine-tuning approaches for BERT. Our experiments suggest that FinBERT-QA, a model built from applying the Transfer and Adapt further fine-tuning and pointwise learning approach, is the most effective, improving the state-of-the-art results of task 2 of the FiQA dataset by 16% on MRR, 17% on NDCG, and 21% on Precision@1.