Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors

TL;DR

This paper introduces an unsupervised method to rank and evaluate the disentanglement of generative models based on training dynamics, without requiring knowledge of latent factors, aiding model selection and downstream task performance.

Contribution

The authors propose a novel, label-free approach to assess disentanglement in generative models using training dynamics, bridging a gap in model evaluation techniques.

Findings

The method correlates with supervised disentanglement metrics.

It effectively predicts downstream performance in reinforcement learning.

It provides a label-free evaluation of model quality.

Abstract

Probabilistic generative models provide a flexible and systematic framework for learning the underlying geometry of data. However, model selection in this setting is challenging, particularly when selecting for ill-defined qualities such as disentanglement or interpretability. In this work, we address this gap by introducing a method for ranking generative models based on the training dynamics exhibited during learning. Inspired by recent theoretical characterizations of disentanglement, our method does not require supervision of the underlying latent factors. We evaluate our approach by demonstrating the need for disentanglement metrics which do not require labels\textemdash the underlying generative factors. We additionally demonstrate that our approach correlates with baseline supervised methods for evaluating disentanglement. Finally, we show that our method can be used as an unsupervised indicator for downstream performance on reinforcement learning and fairness-classification problems.