Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts

TL;DR

This paper evaluates the robustness of multimodal large language models in verifying scientific claims from tables and charts, revealing strengths with tables but significant challenges with charts, and highlighting the need for improved multimodal reasoning.

Contribution

The study adapts existing scientific datasets for multimodal claim verification and systematically assesses 12 models, uncovering key gaps in chart understanding and cross-modal generalization.

Findings

Models perform better with table evidence.

Models struggle with chart evidence.

Humans maintain high performance across formats.

Abstract

With the growing number of submitted scientific papers, there is an increasing demand for systems that can assist reviewers in evaluating research claims. Experimental results are a core component of scientific work, often presented in varying formats such as tables or charts. Understanding how robust current multimodal large language models (multimodal LLMs) are at verifying scientific claims across different evidence formats remains an important and underexplored challenge. In this paper, we design and conduct a series of experiments to assess the ability of multimodal LLMs to verify scientific claims using both tables and charts as evidence. To enable this evaluation, we adapt two existing datasets of scientific papers by incorporating annotations and structures necessary for a multimodal claim verification task. Using this adapted dataset, we evaluate 12 multimodal LLMs and find that current models perform better with table-based evidence while struggling with chart-based evidence. We further conduct human evaluations and observe that humans maintain strong performance across both formats, unlike the models. Our analysis also reveals that smaller multimodal LLMs (under 8B) show weak correlation in performance between table-based and chart-based tasks, indicating limited cross-modal generalization. These findings highlight a critical gap in current models' multimodal reasoning capabilities. We suggest that future multimodal LLMs should place greater emphasis on improving chart understanding to better support scientific claim verification.