VisFinEval: A Scenario-Driven Chinese Multimodal Benchmark for Holistic Financial Understanding

TL;DR

VisFinEval introduces a comprehensive Chinese multimodal benchmark for financial understanding, evaluating 21 models across diverse financial tasks and modalities, revealing strengths and areas for improvement in domain-specific multimodal models.

Contribution

This paper presents the first large-scale Chinese multimodal financial benchmark, VisFinEval, with extensive annotations and scenario-based organization to evaluate and advance financial multimodal large language models.

Findings

Qwen-VL-max achieves 76.3% accuracy in zero-shot evaluation.

Models outperform non-expert humans but lag behind financial experts.

Identifies key failure modes like misalignment and hallucinations.

Abstract

Multimodal large language models (MLLMs) hold great promise for automating complex financial analysis. To comprehensively evaluate their capabilities, we introduce VisFinEval, the first large-scale Chinese benchmark that spans the full front-middle-back office lifecycle of financial tasks. VisFinEval comprises 15,848 annotated question-answer pairs drawn from eight common financial image modalities (e.g., K-line charts, financial statements, official seals), organized into three hierarchical scenario depths: Financial Knowledge & Data Analysis, Financial Analysis & Decision Support, and Financial Risk Control & Asset Optimization. We evaluate 21 state-of-the-art MLLMs in a zero-shot setting. The top model, Qwen-VL-max, achieves an overall accuracy of 76.3%, outperforming non-expert humans but trailing financial experts by over 14 percentage points. Our error analysis uncovers six recurring failure modes-including cross-modal misalignment, hallucinations, and lapses in business-process reasoning-that highlight critical avenues for future research. VisFinEval aims to accelerate the development of robust, domain-tailored MLLMs capable of seamlessly integrating textual and visual financial information. The data and the code are available at https://github.com/SUFE-AIFLM-Lab/VisFinEval.