Evaluating Disentanglement of Structured Representations

TL;DR

This paper presents a new metric for evaluating disentanglement in structured latent representations, especially in object-centric models, providing stronger guarantees and addressing limitations of previous metrics.

Contribution

The paper introduces the first metric for hierarchical disentanglement evaluation and a probing algorithm that handles slot permutation invariance in object-centric models.

Findings

The new metric offers stronger theoretical guarantees for model selection.

Viewing object compositionality as disentanglement improves evaluation accuracy.

The probing algorithm effectively manages slot permutation invariance.

Abstract

We introduce the first metric for evaluating disentanglement at individual hierarchy levels of a structured latent representation. Applied to object-centric generative models, this offers a systematic, unified approach to evaluating (i) object separation between latent slots (ii) disentanglement of object properties inside individual slots (iii) disentanglement of intrinsic and extrinsic object properties. We theoretically show that for structured representations, our framework gives stronger guarantees of selecting a good model than previous disentanglement metrics. Experimentally, we demonstrate that viewing object compositionality as a disentanglement problem addresses several issues with prior visual metrics of object separation. As a core technical component, we present the first representation probing algorithm handling slot permutation invariance.