MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application

TL;DR

MultiFinBen is a comprehensive benchmark for evaluating large language models in realistic financial scenarios involving multiple languages and modalities, revealing significant gaps in current model capabilities.

Contribution

It introduces the first expert-annotated multilingual and multimodal financial benchmark with new tasks and a structured evaluation approach, highlighting current model limitations.

Findings

Frontier models like GPT-4o achieve only 46% overall performance.

Models perform better on vision and audio tasks than multilingual reasoning.

Persistent challenges remain in multilingual, multimodal, and expert-level financial reasoning.

Abstract

Real-world financial analysis involves information across multiple languages and modalities, from reports and news to scanned filings and meeting recordings. Yet most existing evaluations of LLMs in finance remain text-only, monolingual, and largely saturated by current models. To bridge these gaps, we present MultiFinBen, the first expert-annotated multilingual (five languages) and multimodal (text, vision, audio) benchmark for evaluating LLMs in realistic financial contexts. MultiFinBen introduces two new task families: multilingual financial reasoning, which tests cross-lingual evidence integration from filings and news, and financial OCR, which extracts structured text from scanned documents containing tables and charts. Rather than aggregating all available datasets, we apply a structured, difficulty-aware selection based on advanced model performance, ensuring balanced challenge and removing redundant tasks. Evaluating 21 leading LLMs shows that even frontier multimodal models like GPT-4o achieve only 46.01% overall, stronger on vision and audio but dropping sharply in multilingual settings. These findings expose persistent limitations in multilingual, multimodal, and expert-level financial reasoning. All datasets, evaluation scripts, and leaderboards are publicly released.