InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information

TL;DR

InterChart is a comprehensive benchmark designed to evaluate vision-language models' ability to reason across multiple related charts, highlighting their limitations in complex, multi-visual reasoning tasks.

Contribution

The paper introduces InterChart, a new benchmark with diverse, multi-tiered chart reasoning tasks to systematically assess and challenge current vision-language models.

Findings

Models show significant accuracy drops with increased chart complexity.

Decomposing multi-entity charts into simpler units improves model performance.

InterChart exposes systematic limitations in current multimodal reasoning models.

Abstract

We introduce InterChart, a diagnostic benchmark that evaluates how well vision-language models (VLMs) reason across multiple related charts, a task central to real-world applications such as scientific reporting, financial analysis, and public policy dashboards. Unlike prior benchmarks focusing on isolated, visually uniform charts, InterChart challenges models with diverse question types ranging from entity inference and trend correlation to numerical estimation and abstract multi-step reasoning grounded in 2-3 thematically or structurally related charts. We organize the benchmark into three tiers of increasing difficulty: (1) factual reasoning over individual charts, (2) integrative analysis across synthetically aligned chart sets, and (3) semantic inference over visually complex, real-world chart pairs. Our evaluation of state-of-the-art open- and closed-source VLMs reveals consistent and steep accuracy declines as chart complexity increases. We find that models perform better when we decompose multi-entity charts into simpler visual units, underscoring their struggles with cross-chart integration. By exposing these systematic limitations, InterChart provides a rigorous framework for advancing multimodal reasoning in complex, multi-visual environments.