FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation

TL;DR

FinMTM introduces a comprehensive multi-turn multimodal benchmark for financial reasoning, addressing limitations of existing single-turn, narrow-format datasets, and evaluates 22 vision-language models on diverse, realistic financial tasks.

Contribution

The paper presents FinMTM, a novel benchmark with extensive bilingual financial QA pairs and diverse task formats, enabling more realistic evaluation of VLMs in finance.

Findings

VLMs show limitations in visual perception and reasoning.

Multi-turn dialogues reveal challenges in long-context understanding.

Agent-based tasks expose gaps in complex workflow reasoning.

Abstract

The financial domain poses substantial challenges for vision-language models (VLMs) due to specialized chart formats and knowledge-intensive reasoning requirements. However, existing financial benchmarks are largely single-turn and rely on a narrow set of question formats, limiting comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11{,}133 bilingual (Chinese and English) financial QA pairs grounded in financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single- and multiple-choice questions, multi-turn open-ended dialogues, and agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple-choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context reasoning, and complex agent workflows.