WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain

TL;DR

This paper introduces FLANG, a domain-specific financial language model that leverages financial terminology for improved performance, along with the FLUE benchmark suite for comprehensive evaluation across multiple financial NLP tasks.

Contribution

The paper presents a novel financial language model (FLANG) utilizing financial keywords and new training objectives, and introduces the FLUE benchmark suite for extensive evaluation.

Findings

FLANG outperforms previous models on various financial NLP tasks.

The FLUE benchmark provides a comprehensive evaluation framework for financial language understanding.

Open-source release of models, code, and benchmarks facilitates future research.

Abstract

Pre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. Our models, code and benchmark data are publicly available on Github and Huggingface.