Visual symbolic mechanisms: Emergent symbol processing in vision language models

TL;DR

This paper uncovers emergent symbolic mechanisms in vision language models that support object binding through spatial indexing, explaining their failures and suggesting ways to improve their accuracy.

Contribution

It reveals a novel, content-independent spatial indexing mechanism in VLMs that underpins symbol-like processing and binding, previously uncharacterized in such models.

Findings

VLMs employ a content-independent spatial indexing scheme for binding.

Binding errors are linked to failures in these symbolic mechanisms.

Understanding these mechanisms offers pathways to reduce binding failures.

Abstract

To accurately process a visual scene, observers must bind features together to represent individual objects. This capacity is necessary, for instance, to distinguish an image containing a red square and a blue circle from an image containing a blue square and a red circle. Recent work has found that language models solve this 'binding problem' via a set of symbol-like, content-independent indices, but it is unclear whether similar mechanisms are employed by Vision Language Models (VLMs). This question is especially relevant, given the persistent failures of VLMs on tasks that require binding. Here, we identify a previously unknown set of emergent symbolic mechanisms that support binding specifically in VLMs, via a content-independent, spatial indexing scheme. Moreover, we find that binding errors, when they occur, can be traced directly to failures in these mechanisms. Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for reducing the number of binding failures exhibited by these models.