Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies

TL;DR

This paper introduces VIA-Bench, a new benchmark for testing multimodal large language models on visual illusions and anomalies, revealing significant vulnerabilities and highlighting the gap between machine and human perception.

Contribution

The paper presents VIA-Bench, a comprehensive benchmark with over 1,000 questions to evaluate MLLMs on visual illusions, exposing their weaknesses and the limited robustness of Chain-of-Thought reasoning.

Findings

MLLMs show significant vulnerabilities on visual illusions.

Chain-of-Thought reasoning offers negligible robustness.

Models often fail under illusory stimuli, unlike humans.

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable proficiency on general-purpose vision-language benchmarks, reaching or even exceeding human-level performance. However, these evaluations typically rely on standard in-distribution data, leaving the robustness of MLLMs largely unexamined when faced with scenarios that defy common-sense priors. To address this gap, we introduce VIA-Bench, a challenging benchmark designed to probe model performance on visual illusions and anomalies. It includes six core categories: color illusions, motion illusions, gestalt illusions, geometric and spatial illusions, general visual illusions, and visual anomalies. Through careful human-in-the-loop review, we construct over 1K high-quality question-answer pairs that require nuanced visual reasoning. Extensive evaluation of over 20 state-of-the-art MLLMs, including proprietary, open-source, and reasoning-enhanced models, uncovers significant vulnerabilities. Notably, we find that Chain-of-Thought (CoT) reasoning offers negligible robustness, often yielding ``brittle mirages'' where the model's logic collapses under illusory stimuli. Our findings reveal a fundamental divergence between machine and human perception, suggesting that resolving such perceptual bottlenecks is critical for the advancement of artificial general intelligence. The benchmark data and code will be released.