Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?

TL;DR

This paper investigates whether large pretrained Vision Transformers naturally develop the ability to bind object features together, finding that this capability emerges through specific pretraining objectives and actively guides attention.

Contribution

It demonstrates that object binding, a key aspect of human cognition, naturally emerges in Vision Transformers trained with certain objectives, challenging previous assumptions.

Findings

Object binding can be decoded with over 90% accuracy from ViT embeddings.

Emergence of object binding depends on pretraining objectives, stronger in DINO, CLIP, and supervised models.

Object binding signals are encoded in a low-dimensional subspace and influence attention mechanisms.

Abstract

Object binding, the brain's ability to bind the many features that collectively represent an object into a coherent whole, is central to human cognition. It groups low-level perceptual features into high-level object representations, stores those objects efficiently and compositionally in memory, and supports human reasoning about individual object instances. While prior work often imposes object-centric attention (e.g., Slot Attention) explicitly to probe these benefits, it remains unclear whether this ability naturally emerges in pre-trained Vision Transformers (ViTs). Intuitively, they could: recognizing which patches belong to the same object should be useful for downstream prediction and thus guide attention. Motivated by the quadratic nature of self-attention, we hypothesize that ViTs represent whether two patches belong to the same object, a property we term IsSameObject. We decode IsSameObject from patch embeddings across ViT layers using a quadratic similarity probe, which reaches over 90% accuracy. Crucially, this object-binding capability emerges reliably in DINO, CLIP, and ImageNet-supervised ViTs, but is markedly weaker in MAE, suggesting that binding is not a trivial architectural artifact, but an ability acquired through specific pretraining objectives. We further discover that IsSameObject is encoded in a low-dimensional subspace on top of object features, and that this signal actively guides attention. Ablating IsSameObject from model activations degrades downstream performance and works against the learning objective, implying that emergent object binding naturally serves the pretraining objective. Our findings challenge the view that ViTs lack object binding and highlight how symbolic knowledge of "which parts belong together" emerges naturally in a connectionist system.