Probing Perceptual Constancy in Large Vision-Language Models

TL;DR

This study evaluates 155 vision-language models on their ability to maintain perceptual constancy across color, size, and shape, revealing significant variability and dissociations in their perceptual stability.

Contribution

It provides a comprehensive assessment of perceptual constancy in large vision-language models using diverse experiments and introduces new in-the-wild tasks.

Findings

Shape constancy performance differs from color and size.

Performance varies significantly across models and domains.

In-the-wild tasks reveal real-world perceptual challenges.

Abstract

Perceptual constancy is the ability to maintain stable perceptions of objects despite changes in sensory input, such as variations in distance, angle, or lighting. This ability is crucial for visual understanding in a dynamic world. Here, we explored such ability in current Vision Language Models (VLMs). In this study, we evaluated 155 VLMs using 236 experiments across three domains: color, size, and shape constancy. The experiments included single-image and video adaptations of classic cognitive tasks, along with novel tasks in in-the-wild conditions. We found significant variability in VLM performance across these domains, with model performance in shape constancy clearly dissociated from that of color and size constancy.