Rethinking Disentanglement under Dependent Factors of Variation

TL;DR

This paper redefines disentanglement in representation learning to account for dependent factors of variation, proposing a new measurement method validated through experiments where traditional metrics fail.

Contribution

It introduces an information-theoretic definition of disentanglement that handles dependent factors and links it to the Information Bottleneck Method.

Findings

Proposed method accurately measures disentanglement with dependent factors.

Traditional metrics fail under dependent factors, while the new method succeeds.

Experiments demonstrate the effectiveness of the proposed approach.

Abstract

Representation learning is an approach that allows to discover and extract the factors of variation from the data. Intuitively, a representation is said to be disentangled if it separates the different factors of variation in a way that is understandable to humans. Definitions of disentanglement and metrics to measure it usually assume that the factors of variation are independent of each other. However, this is generally false in the real world, which limits the use of these definitions and metrics to very specific and unrealistic scenarios. In this paper we give a definition of disentanglement based on information theory that is also valid when the factors of variation are not independent. Furthermore, we relate this definition to the Information Bottleneck Method. Finally, we propose a method to measure the degree of disentanglement from the given definition that works when the factors of variation are not independent. We show through different experiments that the method proposed in this paper correctly measures disentanglement with non-independent factors of variation, while other methods fail in this scenario.