MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

TL;DR

MagiC is a new benchmark for evaluating multimodal cognition in vision-language models, focusing on reasoning quality, visual grounding, and robustness, revealing current limitations and guiding future improvements.

Contribution

The paper introduces MagiC, a comprehensive benchmark with new metrics for assessing grounded visual reasoning and model robustness in vision-language models.

Findings

Models often rely on superficial patterns rather than true reasoning.

Grounding fidelity varies significantly across models.

Benchmark reveals key limitations in current multimodal reasoning approaches.

Abstract

Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual reasoning or rely on superficial patterns and dataset biases. In this work, we introduce MagiC, a comprehensive benchmark designed to evaluate grounded multimodal cognition, assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence. Our benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. We evaluate 15 vision-language models ranging from 7B to 70B parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability. MagiC further includes diagnostic settings to probe model robustness under adversarial visual cues and assess their capacity for introspective error correction. We introduce new metrics such as MagiScore and StepSense, and provide comprehensive analyses that reveal key limitations and opportunities in current approaches to grounded visual reasoning.