Pixels, Patterns, but No Poetry: To See The World like Humans

TL;DR

This paper introduces the Turing Eye Test, a new benchmark for evaluating whether Multimodal Large Language Models perceive the world like humans, revealing current models' perceptual shortcomings despite reasoning strengths.

Contribution

The paper presents the Turing Eye Test, a perception-focused benchmark for MLLMs, highlighting their failures in human-like perception and the importance of vision tower fine-tuning.

Findings

State-of-the-art MLLMs fail on perceptual tasks trivial for humans.

In-context learning and language training do not improve perceptual performance.

Fine-tuning the vision tower enables rapid perceptual adaptation.

Abstract

Achieving human-like perception and reasoning in Multimodal Large Language Models (MLLMs) remains a central challenge in artificial intelligence. While recent research has primarily focused on enhancing reasoning capabilities in MLLMs, a fundamental question persists: Can Multimodal Large Language Models truly perceive the world as humans do? This paper shifts focus from reasoning to perception. Rather than constructing benchmarks specifically for reasoning, we introduce the Turing Eye Test (TET), a challenging perception-oriented benchmark comprising four diagnostic tasks that evaluate MLLMs' performance on synthetic images that humans process intuitively. Our findings reveal that state-of-the-art MLLMs exhibit catastrophic failures on our perceptual tasks trivial for humans. Both in-context learning and training on language backbone-effective for previous benchmarks-fail to improve performance on our tasks, while fine-tuning the vision tower enables rapid adaptation, suggesting that our benchmark poses challenges for vision tower generalization rather than for the knowledge and reasoning capabilities of the language backbone-a key gap between current MLLMs and human perception. We release a representative subset of TET tasks in this version, and will introduce more diverse tasks and methods to enhance visual generalization in future work.