TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models

TL;DR

TopoPerception introduces a topology-based benchmark to evaluate the global visual perception of large vision-language models, revealing their significant limitations and the need for new training methods.

Contribution

The paper presents TopoPerception, a novel, shortcut-free benchmark based on topological properties to assess global visual perception in LVLMs, exposing their perceptual shortcomings.

Findings

All models perform at chance level on global perception tasks.

More powerful models show lower accuracy, indicating increased reasoning ability correlates with perception deficits.

Scaling models alone does not improve global visual perception capabilities.

Abstract

Large Vision-Language Models (LVLMs) typically align visual features from an encoder with a pre-trained Large Language Model (LLM). However, this makes the visual perception module a bottleneck, which constrains the overall capabilities of LVLMs. Conventional evaluation benchmarks, while rich in visual semantics, often contain unavoidable local shortcuts that can lead to an overestimation of models' perceptual abilities. Here, we introduce TopoPerception, a benchmark that leverages topological properties to rigorously evaluate the global visual perception capabilities of LVLMs across various granularities. Since topology depends on the global structure of an image and is invariant to local features, TopoPerception enables a shortcut-free assessment of global perception, fundamentally distinguishing it from semantically rich tasks. We evaluate state-of-the-art models on TopoPerception and find that even at the coarsest perceptual granularity, all models perform no better than random chance, indicating a profound inability to perceive global visual features. Notably, a consistent trend emerge within model families: more powerful models with stronger reasoning capabilities exhibit lower accuracy. This suggests that merely scaling up models is insufficient to address this deficit and may even exacerbate it. Progress may require new training paradigms or architectures. TopoPerception not only exposes a critical bottleneck in current LVLMs but also offers a lens and direction for improving their global visual perception. The data and code are publicly available at: https://github.com/Wenhao-Zhou/TopoPerception.