BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs

TL;DR

BareBones is a new benchmark that rigorously tests whether vision-language models truly understand geometric shapes, revealing widespread reliance on textures and environmental cues.

Contribution

It introduces a pixel-level silhouette benchmark across multiple datasets, exposing the texture bias in current models and establishing a standard for geometric comprehension evaluation.

Findings

Models perform poorly without RGB textures, indicating a reliance on visual shortcuts.

The benchmark exposes universal structural blindspots in state-of-the-art VLMs.

Performance collapse under RGB deprivation is termed the 'Texture Bias Cliff'.

Abstract

While Vision-Language Models (VLMs) demonstrate remarkable zero-shot recognition capabilities across a diverse spectrum of multimodal tasks, it yet remains an open question whether these architectures genuinely comprehend geometric structure or merely exploit RGB textures and contextual priors as statistical shortcuts. Existing evaluations fail to isolate this mechanism, conflating semantic reasoning with texture mapping and relying on imprecise annotations that inadvertently leak environmental cues. To address this gap, we introduce $\textbf{BareBones}$, a zero-shot benchmark designed to stress-test pure geometric shape comprehension. We curate pixel-level silhouettes of geometrically distinct classes across six datasets: five established segmentation sources (ImageNet-S, DIS5K, ThinObject5K, PASCAL VOC, CUB-200) and our novel flagship collection, WTP-Bench, establishing a noise-free geometric taxonomy. WTP-Bench is an extreme, fine-grained visual puzzle that forces models to identify inter-class geometric concepts from boundary contours alone. Our evaluation of 26 state-of-the-art proprietary and open-weight VLMs (eg. GPT-4.1, Gemini, Claude Sonnet 4.5, LLaVA) reveals a consistent, severe performance collapse under RGB deprivation, a phenomenon we term the $\textit{Texture Bias Cliff}$. By documenting universal structural blindspots, BareBones establishes a rigorous yardstick for genuine geometric grounding. Project Page: https://eternal-f1ame.github.io/WTP-Bench/