Are Object-Centric Representations Better At Compositional Generalization?

TL;DR

This paper introduces a new benchmark to evaluate how well object-centric and dense vision encoders generalize to unseen object combinations in visual question answering, revealing that object-centric models excel under data or compute constraints.

Contribution

It provides a systematic comparison of object-centric versus dense representations in compositional generalization across three visual worlds, accounting for various training factors.

Findings

Object-centric models outperform dense models in challenging generalization settings.

Dense representations are better on easier tasks but require more data and compute.

Object-centric models are more sample-efficient and perform better with limited data.

Abstract

Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of dataset size, training data diversity, or downstream compute is constrained.