Successes and Limitations of Object-centric Models at Compositional Generalisation

TL;DR

This paper evaluates object-centric models' ability to generalize compositionally in visual scenes, demonstrating their strengths, limitations, and potential improvements through training, while identifying remaining challenges for future research.

Contribution

It extends understanding of object-centric models' compositional generalization to property combinations, not just scene composition, and explores how training influences their capabilities.

Findings

Object-centric models generalize to novel object property combinations.

Careful training can enhance compositional skills.

Limitations remain, indicating directions for future research.

Abstract

In recent years, it has been shown empirically that standard disentangled latent variable models do not support robust compositional learning in the visual domain. Indeed, in spite of being designed with the goal of factorising datasets into their constituent factors of variations, disentangled models show extremely limited compositional generalisation capabilities. On the other hand, object-centric architectures have shown promising compositional skills, albeit these have 1) not been extensively tested and 2) experiments have been limited to scene composition -- where models must generalise to novel combinations of objects in a visual scene instead of novel combinations of object properties. In this work, we show that these compositional generalisation skills extend to this later setting. Furthermore, we present evidence pointing to the source of these skills and how they can be improved through careful training. Finally, we point to one important limitation that still exists which suggests new directions of research.