Evaluating Object-Centric Models beyond Object Discovery

TL;DR

This paper proposes a comprehensive evaluation framework for object-centric learning models, using instruction-tuned vision-language models and unified metrics to better assess their usefulness for complex reasoning and localization tasks.

Contribution

It introduces a unified evaluation task and metric, and employs instruction-tuned VLMs for scalable benchmarking across diverse reasoning datasets.

Findings

Unified evaluation improves assessment consistency

Instruction-tuned VLMs effectively measure representation usefulness

Baseline provides reference for multi-feature reconstruction

Abstract

Object-centric learning (OCL) aims to learn structured scene representations that support compositional generalization and robustness to out-of-distribution (OOD) data. However, OCL models are often not evaluated regarding these goals. Instead, most prior work focuses on evaluating OCL models solely through object discovery and simple reasoning tasks, such as probing the representation via image classification. We identify two limitations in existing benchmarks: (1) They provide limited insights on the representation usefulness of OCL models, and (2) localization and representation usefulness are assessed using disjoint metrics. To address (1), we use instruction-tuned VLMs as evaluators, enabling scalable benchmarking across diverse VQA datasets to measure how well VLMs leverage OCL representations for complex reasoning tasks. To address (2), we introduce a unified evaluation task and metric that jointly assess localization (where) and representation usefulness (what), thereby eliminating inconsistencies introduced by disjoint evaluation. Finally, we include a simple multi-feature reconstruction baseline as a reference point.