Disentangling Granularity: An Implicit Inductive Bias in Factorized VAEs

TL;DR

This paper uncovers an implicit inductive bias called disentangling granularity in factorized VAEs, which influences their ability to disentangle features of varying complexity, and validates this through extensive experiments.

Contribution

It identifies and analyzes the role of disentangling granularity as an implicit bias in VAEs, and proposes ta-STCVAE to control this bias for improved disentanglement.

Findings

Disentangling granularity affects the complexity of features disentangled by VAEs.

Tuning granularity broadens the range of features that can be disentangled.

Experimental validation with over 100K runs supports the theoretical insights.

Abstract

Despite the success in learning semantically meaningful, unsupervised disentangled representations, variational autoencoders (VAEs) and their variants face a fundamental theoretical challenge: substantial evidence indicates that unsupervised disentanglement is unattainable without implicit inductive bias, yet such bias remains elusive. In this work, we focus on exploring the implicit inductive bias that drive disentanglement in VAEs with factorization priors. By analyzing the total correlation in \b{eta}-TCVAE, we uncover a crucial implicit inductive bias called disentangling granularity, which leads to the discovery of an interesting "V"-shaped optimal Evidence Lower Bound (ELBO) trajectory within the parameter space. This finding is validated through over 100K experiments using factorized VAEs and our newly proposed model, \b{eta}-STCVAE. Notably, experimental results reveal that conventional factorized VAEs, constrained by fixed disentangling granularity, inherently tend to disentangle low-complexity feature. Whereas, appropriately tuning disentangling granularity, as enabled by \b{eta}-STCVAE, broadens the range of disentangled representations, allowing for the disentanglement of high-complexity features. Our findings unveil that disentangling granularity as an implicit inductive bias in factorized VAEs influence both disentanglement performance and the inference of the ELBO, offering fresh insights into the interpretability and inherent biases of VAEs.