Contextual inference from single objects in Vision-Language models

TL;DR

This study investigates how vision-language models infer scene context from single objects, revealing their capabilities, limitations, and underlying mechanisms in understanding scene categories and superordinate contexts.

Contribution

It provides a systematic analysis of contextual inference in VLMs, highlighting differences from human perception and uncovering the mechanistic basis of scene understanding.

Findings

Single objects enable above-chance scene inference in VLMs.

Object properties predict human-like scene categorization.

Scene and superordinate information are encoded differently within models.

Abstract

How much scene context a single object carries is a well-studied question in human scene perception, yet how this capacity is organized in vision-language models (VLMs) remains poorly understood, with direct implications for the robustness of these models. We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects. Presenting VLMs with single objects on masked backgrounds, we probe their ability to infer both fine-grained scene category and coarse superordinate context (indoor vs. outdoor). We found that single objects support above-chance inference at both levels, with performance modulated by the same object properties that predict human scene categorization. Object identity, scene, and superordinate predictions are partially dissociable: accurate inference at one level neither requires nor guarantees accurate inference at the others, and the degree of coupling differs markedly across models. Mechanistically, object representations that remain stable when background context is removed are more predictive of successful contextual inference. Scene and superordinate schemas are grounded in fundamentally different ways: scene identity is encoded in image tokens throughout the network, while superordinate information emerges only late or not at all. Together, these results reveal that the organization of contextual inference in VLMs is more complex than accuracy alone suggests, with behavioral and mechanistic signatures