Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication

TL;DR

This paper introduces the COOCo dataset to study how vision-language models utilize scene context and semantic violations in object referencing, revealing adaptive reliance on context and attention patterns related to semantic fit.

Contribution

The paper presents the COOCo dataset and analyzes VLMs' reliance on scene context and attention dynamics in object referencing tasks.

Findings

Models adaptively use scene context based on semantic relatedness and noise.

Successful categorization correlates with increased mid-layer attention to the target.

Attention patterns vary non-monotonically with semantic fit, dropping at moderate fit and rising at low and high fit.

Abstract

To what degree and under what conditions do VLMs rely on scene context when generating references to objects? To address this question, we introduce the $\textit{Common Objects Out-of-Context (COOCo)}$ dataset and conduct experiments on several VLMs under different degrees of scene-object congruency and noise. We find that models leverage scene context adaptively, depending on scene-object semantic relatedness and noise level. Based on these consistent trends across models, we turn to the question of how VLM attention patterns change as a function of target-scene semantic fit, and to what degree these patterns are predictive of categorisation accuracy. We find that successful object categorisation is associated with increased mid-layer attention to the target. We also find a non-monotonic dependency on semantic fit, with attention dropping at moderate fit and increasing for both low and high fit. These results suggest that VLMs dynamically balance local and contextual information for reference generation. Dataset and code are available here: $\href{https://github.com/cs-nlp-uu/scenereg}{https://github.com/cs-nlp-uu/scenereg}$.