Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMs

TL;DR

This paper introduces Hyperphantasia, a benchmark to evaluate the mental visualization abilities of multimodal large language models, revealing significant gaps compared to human performance and exploring reinforcement learning improvements.

Contribution

The paper presents Hyperphantasia, a novel benchmark with procedurally generated puzzles to assess active visual construction in MLLMs, addressing a key gap in current evaluation methods.

Findings

State-of-the-art models perform significantly worse than humans in visualization tasks.

Reinforcement learning shows potential to enhance visual simulation capabilities.

Robust mental visualization remains a major challenge for current MLLMs.

Abstract

Mental visualization, the ability to construct and manipulate visual representations internally, is a core component of human cognition and plays a vital role in tasks involving reasoning, prediction, and abstraction. Despite the rapid progress of Multimodal Large Language Models (MLLMs), current benchmarks primarily assess passive visual perception, offering limited insight into the more active capability of internally constructing visual patterns to support problem solving. Yet mental visualization is a critical cognitive skill in humans, supporting abilities such as spatial navigation, predicting physical trajectories, and solving complex visual problems through imaginative simulation. To bridge this gap, we introduce Hyperphantasia, a synthetic benchmark designed to evaluate the mental visualization abilities of MLLMs through four carefully constructed puzzles. Each puzzle is procedurally generated and presented at three difficulty levels, enabling controlled analysis of model performance across increasing complexity. Our comprehensive evaluation of state-of-the-art models reveals a substantial gap between the performance of humans and MLLMs. Additionally, we explore the potential of reinforcement learning to improve visual simulation capabilities. Our findings suggest that while some models exhibit partial competence in recognizing visual patterns, robust mental visualization remains an open challenge for current MLLMs.