Disentanglement Analysis with Partial Information Decomposition

TL;DR

This paper introduces a new framework using Partial Information Decomposition to analyze how multivariate representations disentangle generative factors, addressing limitations of existing metrics by capturing complex entanglements involving multiple variables.

Contribution

It proposes a novel information-theoretic framework and metric for disentanglement analysis that considers multivariate interactions beyond pairwise relationships.

Findings

The new metric correctly detects entanglement in representations.

Models with similar scores can have different entanglement characteristics.

Different strategies may be needed to achieve disentanglement based on entanglement types.

Abstract

We propose a framework to analyze how multivariate representations disentangle ground-truth generative factors. A quantitative analysis of disentanglement has been based on metrics designed to compare how one variable explains each generative factor. Current metrics, however, may fail to detect entanglement that involves more than two variables, e.g., representations that duplicate and rotate generative factors in high dimensional spaces. In this work, we establish a framework to analyze information sharing in a multivariate representation with Partial Information Decomposition and propose a new disentanglement metric. This framework enables us to understand disentanglement in terms of uniqueness, redundancy, and synergy. We develop an experimental protocol to assess how increasingly entangled representations are evaluated with each metric and confirm that the proposed metric correctly responds to entanglement. Through experiments on variational autoencoders, we find that models with similar disentanglement scores have a variety of characteristics in entanglement, for each of which a distinct strategy may be required to obtain a disentangled representation.