CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

TL;DR

This paper introduces CFBenchmark-MM, a Chinese multimodal financial benchmark with over 9,000 image-question pairs, to evaluate and improve multimodal large language models in financial analysis, highlighting current limitations and future needs.

Contribution

The paper presents a new Chinese multimodal financial benchmark and a staged evaluation system for assessing MLLMs' handling of multimodal financial data.

Findings

MLLMs show limited efficiency in financial multimodal tasks.

Misinterpretation of visual content is a primary challenge.

Further domain-specific optimization is needed for better performance.

Abstract

Multimodal Large Language Models (MLLMs) have rapidly evolved with the growth of Large Language Models (LLMs) and are now applied in various fields. In finance, the integration of diverse modalities such as text, charts, and tables is crucial for accurate and efficient decision-making. Therefore, an effective evaluation system that incorporates these data types is essential for advancing financial application. In this paper, we introduce CFBenchmark-MM, a Chinese multimodal financial benchmark with over 9,000 image-question pairs featuring tables, histogram charts, line charts, pie charts, and structural diagrams. Additionally, we develop a staged evaluation system to assess MLLMs in handling multimodal information by providing different visual content step by step. Despite MLLMs having inherent financial knowledge, experimental results still show limited efficiency and robustness in handling multimodal financial context. Further analysis on incorrect responses reveals the misinterpretation of visual content and the misunderstanding of financial concepts are the primary issues. Our research validates the significant, yet underexploited, potential of MLLMs in financial analysis, highlighting the need for further development and domain-specific optimization to encourage the enhanced use in financial domain.