MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems

TL;DR

MultiChartQA is a new benchmark designed to evaluate multimodal large language models on complex multi-chart reasoning tasks, addressing gaps in existing single-chart focused benchmarks and highlighting current performance limitations.

Contribution

The paper introduces MultiChartQA, a comprehensive benchmark for multi-chart reasoning in vision-language models, emphasizing multi-hop, comparative, and sequential reasoning capabilities.

Findings

MLLMs show significant performance gaps compared to humans.

Current models struggle with multi-chart multi-hop reasoning.

MultiChartQA reveals key challenges in multi-chart comprehension.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive abilities across various tasks, including visual question answering and chart comprehension, yet existing benchmarks for chart-related tasks fall short in capturing the complexity of real-world multi-chart scenarios. Current benchmarks primarily focus on single-chart tasks, neglecting the multi-hop reasoning required to extract and integrate information from multiple charts, which is essential in practical applications. To fill this gap, we introduce MultiChartQA, a benchmark that evaluates MLLMs' capabilities in four key areas: direct question answering, parallel question answering, comparative reasoning, and sequential reasoning. Our evaluation of a wide range of MLLMs reveals significant performance gaps compared to humans. These results highlight the challenges in multi-chart comprehension and the potential of MultiChartQA to drive advancements in this field. Our code and data are available at https://github.com/Zivenzhu/Multi-chart-QA