DEFT: Distilling Entangled Factors by Preventing Information Diffusion

TL;DR

DEFT introduces a novel framework to improve disentanglement in representation learning by addressing the information diffusion problem through staged training and information scaling, achieving high-quality disentangled factors.

Contribution

The paper proposes DEFT, a new disentanglement method that prevents information diffusion by scaling backward information, with a multistage training strategy for better factor separation.

Findings

DEFT outperforms existing methods on dSprite and SmallNORB datasets.

DEFT achieves low-variance and high-level disentanglement scores.

DEFT is effective in unsupervised disentanglement scenarios.

Abstract

Disentanglement is a highly desirable property of representation owing to its similarity to human understanding and reasoning. Many works achieve disentanglement upon information bottlenecks (IB). Despite their elegant mathematical foundations, the IB branch usually exhibits lower performance. In order to provide an insight into the problem, we develop an annealing test to calculate the information freezing point (IFP), which is a transition state to freeze information into the latent variables. We also explore these clues or inductive biases for separating the entangled factors according to the differences in the IFP distributions. We found the existing approaches suffer from the information diffusion problem, according to which the increased information diffuses in all latent variables. Based on this insight, we propose a novel disentanglement framework, termed the distilling entangled factor (DEFT), to address the information diffusion problem by scaling backward information. DEFT applies a multistage training strategy, including multigroup encoders with different learning rates and piecewise disentanglement pressure, to disentangle the factors stage by stage. We evaluate DEFT on three variants of dSprite and SmallNORB, which show low-variance and high-level disentanglement scores. Furthermore, the experiment under the correlative factors shows incapable of TC-based approaches. DEFT also exhibits a competitive performance in the unsupervised setting.