MMTBENCH: A Unified Benchmark for Complex Multimodal Table Reasoning

TL;DR

MMTBENCH is a comprehensive benchmark with 500 real-world multimodal tables designed to evaluate and advance vision-language models' ability to perform complex reasoning involving visual and tabular data.

Contribution

This paper introduces MMTBENCH, the first large-scale benchmark for complex multimodal table reasoning, highlighting current models' performance gaps and guiding future research.

Findings

State-of-the-art models perform poorly on visual-based and multi-step reasoning questions.

Significant performance gaps exist across different question and table types.

The benchmark reveals the need for better integration of vision and language in models.

Abstract

Multimodal tables those that integrate semi structured data with visual elements such as charts and maps are ubiquitous across real world domains, yet they pose a formidable challenge to current vision language models (VLMs). While Large Language models (LLMs) and VLMs have demonstrated strong capabilities in text and image understanding, their performance on complex, real world multimodal table reasoning remains unexplored. To bridge this gap, we introduce MMTBENCH (Multimodal Table Benchmark), a benchmark consisting of 500 real world multimodal tables drawn from diverse real world sources, with a total of 4021 question answer pairs. MMTBENCH questions cover four question types (Explicit, Implicit, Answer Mention, and Visual Based), five reasoning types (Mathematical, Extrema Identification, Fact Verification, Vision Based, and Others), and eight table types (Single/Multiple Entity, Maps and Charts with Entities, Single/Multiple Charts, Maps, and Visualizations). Extensive evaluation of state of the art models on all types reveals substantial performance gaps, particularly on questions requiring visual-based reasoning and multi-step inference. These findings show the urgent need for improved architectures that more tightly integrate vision and language processing. By providing a challenging, high-quality resource that mirrors the complexity of real-world tasks, MMTBENCH underscores its value as a resource for future research on multimodal tables.