NumLLM: Numeric-Sensitive Large Language Model for Chinese Finance

TL;DR

NumLLM is a specialized Chinese financial language model that significantly improves understanding of numeric information in financial texts by using a novel training approach with domain-specific corpora and LoRA modules.

Contribution

The paper introduces NumLLM, a novel numeric-sensitive LLM for Chinese finance, utilizing a financial corpus and dual LoRA modules for enhanced numeric and domain understanding.

Findings

NumLLM outperforms all baselines on financial QA benchmarks.

It significantly improves numeric question understanding.

The approach enhances general and numeric-specific capabilities of LLMs.

Abstract

Recently, many works have proposed various financial large language models (FinLLMs) by pre-training from scratch or fine-tuning open-sourced LLMs on financial corpora. However, existing FinLLMs exhibit unsatisfactory performance in understanding financial text when numeric variables are involved in questions. In this paper, we propose a novel LLM, called numeric-sensitive large language model (NumLLM), for Chinese finance. We first construct a financial corpus from financial textbooks which is essential for improving numeric capability of LLMs during fine-tuning. After that, we train two individual low-rank adaptation (LoRA) modules by fine-tuning on our constructed financial corpus. One module is for adapting general-purpose LLMs to financial domain, and the other module is for enhancing the ability of NumLLM to understand financial text with numeric variables. Lastly, we merge the two LoRA modules into the foundation model to obtain NumLLM for inference. Experiments on financial question-answering benchmark show that NumLLM can boost the performance of the foundation model and can achieve the best overall performance compared to all baselines, on both numeric and non-numeric questions.