FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging

TL;DR

FinMMR is a comprehensive bilingual multimodal benchmark designed to evaluate and advance the numerical reasoning abilities of large language models in complex financial tasks involving diverse data types and subdomains.

Contribution

It introduces a novel multimodal, comprehensive, and challenging benchmark with 4.3K questions and 8.7K images across 14 financial categories, focusing on multi-step reasoning.

Findings

Best model achieves 53.0% accuracy on hard problems

Benchmark covers 14 financial subdomains and multiple data modalities

FinMMR surpasses existing benchmarks in scope and difficulty

Abstract

We present FinMMR, a novel bilingual multimodal benchmark tailored to evaluate the reasoning capabilities of multimodal large language models (MLLMs) in financial numerical reasoning tasks. Compared to existing benchmarks, our work introduces three significant advancements. (1) Multimodality: We meticulously transform existing financial reasoning benchmarks, and construct novel questions from the latest Chinese financial research reports. FinMMR comprises 4.3K questions and 8.7K images spanning 14 categories, including tables, bar charts, and ownership structure charts. (2) Comprehensiveness: FinMMR encompasses 14 financial subdomains, including corporate finance, banking, and industry analysis, significantly exceeding existing benchmarks in financial domain knowledge breadth. (3) Challenge: Models are required to perform multi-step precise numerical reasoning by integrating financial knowledge with the understanding of complex financial images and text. The best-performing MLLM achieves only 53.0% accuracy on Hard problems. We believe that FinMMR will drive advancements in enhancing the reasoning capabilities of MLLMs in real-world scenarios.