Vision-language models learn the geometry of human perceptual space

TL;DR

This paper demonstrates that vision-language models learn a perceptual space similar to human cognition, and this geometry can predict human categorization better than direct human judgments, bridging AI and cognitive science.

Contribution

Introduces a method to analyze the internal geometry of VLMs and shows they capture human-like perceptual structures relevant for categorization.

Findings

VLMs recover low-dimensional perceptual spaces aligned with human perception.

AI-derived spaces predict human categorization more accurately than human judgments.

Provides a scalable approach to study cognitive representations using AI models.

Abstract

In cognitive science and AI, a longstanding question is whether machines learn representations that align with those of the human mind. While current models show promise, it remains an open question whether this alignment is superficial or reflects a deeper correspondence in the underlying dimensions of representation. Here we introduce a methodology to probe the internal geometry of vision-language models (VLMs) by having them generate pairwise similarity judgments for a complex set of natural objects. Using multidimensional scaling, we recover low-dimensional psychological spaces and find that their axes show a strong correspondence with the principal axes of human perceptual space. Critically, when this AI-derived representational geometry is used as the input to a classic exemplar model of categorization, it predicts human classification behavior more accurately than a space constructed from human judgments themselves. This suggests that VLMs can capture an idealized or `denoised' form of human perceptual structure. Our work provides a scalable method to overcome a measurement bottleneck in cognitive science and demonstrates that foundation models can learn a representational geometry that is functionally relevant for modeling key aspects of human cognition, such as categorization.