FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models

TL;DR

FinChart-Bench is a new benchmark dataset designed to evaluate the ability of vision-language models to understand complex financial charts, revealing current limitations in model performance and reasoning skills.

Contribution

This paper introduces FinChart-Bench, the first specialized benchmark for financial chart comprehension in vision-language models, with extensive annotations and evaluation of 25 models.

Findings

Performance gap between open-source and closed-source models is decreasing.

Models show performance degradation in upgraded versions.

Many models struggle with spatial reasoning and instruction following.

Abstract

Large vision-language models (LVLMs) have made significant progress in chart understanding. However, financial charts, characterized by complex temporal structures and domain-specific terminology, remain notably underexplored. We introduce FinChart-Bench, the first benchmark specifically focused on real-world financial charts. FinChart-Bench comprises 1,200 financial chart images collected from 2015 to 2024, each annotated with True/False (TF), Multiple Choice (MC), and Question Answering (QA) questions, totaling 7,016 questions. We conduct a comprehensive evaluation of 25 state-of-the-art LVLMs on FinChart-Bench. Our evaluation reveals critical insights: (1) the performance gap between open-source and closed-source models is narrowing, (2) performance degradation occurs in upgraded models within families, (3) many models struggle with instruction following, (4) both advanced models show significant limitations in spatial reasoning abilities, and (5) current LVLMs are not reliable enough to serve as automated evaluators. These findings highlight important limitations in current LVLM capabilities for financial chart understanding. The FinChart-Bench dataset is available at https://huggingface.co/datasets/Tizzzzy/FinChart-Bench.