BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

TL;DR

BankMathBench is a new domain-specific benchmark dataset designed to evaluate and improve large language models' numerical reasoning abilities in realistic banking scenarios, covering multi-step calculations and multi-condition reasoning.

Contribution

The paper introduces BankMathBench, a structured dataset with three difficulty levels, specifically targeting banking tasks, and demonstrates its effectiveness in enhancing LLMs' numerical reasoning through tool-augmented fine-tuning.

Findings

Significant accuracy improvements in LLMs after training on BankMathBench.

Models achieved up to 75.1 percentage points increase in intermediate difficulty tasks.

BankMathBench provides a reliable benchmark for real-world banking numerical reasoning.

Abstract

Large language models (LLMs)-based chatbots are increasingly being adopted in the financial domain, particularly in digital banking, to handle customer inquiries about products such as deposits, savings, and loans. However, these models still exhibit low accuracy in core banking computations-including total payout estimation, comparison of products with varying interest rates, and interest calculation under early repayment conditions. Such tasks require multi-step numerical reasoning and contextual understanding of banking products, yet existing LLMs often make systematic errors-misinterpreting product types, applying conditions incorrectly, or failing basic calculations involving exponents and geometric progressions. However, such errors have rarely been captured by existing benchmarks. Mathematical datasets focus on fundamental math problems, whereas financial benchmarks primarily target financial documents, leaving everyday banking scenarios underexplored. To address this limitation, we propose BankMathBench, a domain-specific dataset that reflects realistic banking tasks. BankMathBench is organized in three levels of difficulty-basic, intermediate, and advanced-corresponding to single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. When trained on BankMathBench, open-source LLMs exhibited notable improvements in both formula generation and numerical reasoning accuracy, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning. With tool-augmented fine-tuning, the models achieved average accuracy increases of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced), representing significant gains over zero-shot baselines. These findings highlight BankMathBench as a reliable benchmark for evaluating and advancing LLMs' numerical reasoning in real-world banking scenarios.