FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging

TL;DR

FinanceReasoning is a comprehensive benchmark for evaluating large reasoning models' financial numerical reasoning, emphasizing credibility, coverage of financial concepts, and challenging multi-formula problems to advance domain-specific reasoning capabilities.

Contribution

The paper introduces FinanceReasoning, a new benchmark with updated questions, extensive financial concepts, and complex multi-formula problems to better evaluate and improve financial reasoning in large models.

Findings

Models achieve 89.1% accuracy on hard problems.

Refined evaluation standards improve assessment accuracy.

Combining Reasoner and Programmer models enhances performance.

Abstract

We introduce FinanceReasoning, a novel benchmark designed to evaluate the reasoning capabilities of large reasoning models (LRMs) in financial numerical reasoning problems. Compared to existing benchmarks, our work provides three key advancements. (1) Credibility: We update 15.6% of the questions from four public datasets, annotating 908 new questions with detailed Python solutions and rigorously refining evaluation standards. This enables an accurate assessment of the reasoning improvements of LRMs. (2) Comprehensiveness: FinanceReasoning covers 67.8% of financial concepts and formulas, significantly surpassing existing datasets. Additionally, we construct 3,133 Python-formatted functions, which enhances LRMs' financial reasoning capabilities through refined knowledge (e.g., 83.2% $\rightarrow$ 91.6% for GPT-4o). (3) Challenge: Models are required to apply multiple financial formulas for precise numerical reasoning on 238 Hard problems. The best-performing model (i.e., OpenAI o1 with PoT) achieves 89.1% accuracy, yet LRMs still face challenges in numerical precision. We demonstrate that combining Reasoner and Programmer models can effectively enhance LRMs' performance (e.g., 83.2% $\rightarrow$ 87.8% for DeepSeek-R1). Our work paves the way for future research on evaluating and improving LRMs in domain-specific complex reasoning tasks.