Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers

TL;DR

This study evaluates the ability of multimodal foundation models to interpret schematic diagrams in scientific papers through the new MISS-QA benchmark, revealing significant gaps compared to human experts and providing insights for future improvements.

Contribution

Introduces MISS-QA, the first benchmark for assessing models' understanding of schematic diagrams in scientific literature, and evaluates 18 state-of-the-art models on this task.

Findings

Models perform significantly worse than humans on MISS-QA.

Analysis highlights current models' limitations in understanding scientific diagrams.

Error analysis provides insights for future model improvements.

Abstract

This paper introduces MISS-QA, the first benchmark specifically designed to evaluate the ability of models to interpret schematic diagrams within scientific literature. MISS-QA comprises 1,500 expert-annotated examples over 465 scientific papers. In this benchmark, models are tasked with interpreting schematic diagrams that illustrate research overviews and answering corresponding information-seeking questions based on the broader context of the paper. We assess the performance of 18 frontier multimodal foundation models, including o4-mini, Gemini-2.5-Flash, and Qwen2.5-VL. We reveal a significant performance gap between these models and human experts on MISS-QA. Our analysis of model performance on unanswerable questions and our detailed error analysis further highlight the strengths and limitations of current models, offering key insights to enhance models in comprehending multimodal scientific literature.