Evaluating the Disentanglement of Deep Generative Models through Manifold Topology

TL;DR

This paper introduces a novel, model-only method for quantifying disentanglement in deep generative models by analyzing the topology of their learned representations, applicable in both supervised and unsupervised settings.

Contribution

The paper proposes a topological similarity-based metric for disentanglement that does not rely on external models or dataset-specific assumptions, improving measurement consistency.

Findings

The method effectively evaluates and ranks state-of-the-art models.

It aligns well with existing disentanglement metrics.

The approach is applicable across multiple datasets.

Abstract

Learning disentangled representations is regarded as a fundamental task for improving the generalization, robustness, and interpretability of generative models. However, measuring disentanglement has been challenging and inconsistent, often dependent on an ad-hoc external model or specific to a certain dataset. To address this, we present a method for quantifying disentanglement that only uses the generative model, by measuring the topological similarity of conditional submanifolds in the learned representation. This method showcases both unsupervised and supervised variants. To illustrate the effectiveness and applicability of our method, we empirically evaluate several state-of-the-art models across multiple datasets. We find that our method ranks models similarly to existing methods. We make ourcode publicly available at https://github.com/stanfordmlgroup/disentanglement.