Break The Spell Of Total Correlation In betaTCVAE

TL;DR

This paper introduces a novel method for disentangled representation learning in VAEs by iteratively decomposing total correlation, allowing flexible feature separation and improved disentangling performance without labels.

Contribution

It proposes a new objective function that adjusts model capacity to better separate dependent and independent features in unsupervised learning.

Findings

Model capacity correlates with latent variable grouping size.

The method achieves better disentangling performance.

The ELBO trajectory exhibits a 'V'-shaped pattern with capacity changes.

Abstract

In the absence of artificial labels, the independent and dependent features in the data are cluttered. How to construct the inductive biases of the model to flexibly divide and effectively contain features with different complexity is the main focal point of unsupervised disentangled representation learning. This paper proposes a new iterative decomposition path of total correlation and explains the disentangled representation ability of VAE from the perspective of model capacity allocation. The newly developed objective function combines latent variable dimensions into joint distribution while relieving the independence constraints of marginal distributions in combination, leading to latent variables with a more manipulable prior distribution. The novel model enables VAE to adjust the parameter capacity to divide dependent and independent data features flexibly. Experimental results on various datasets show an interesting relevance between model capacity and the latent variable grouping size, called the "V"-shaped best ELBO trajectory. Additionally, we empirically demonstrate that the proposed method obtains better disentangling performance with reasonable parameter capacity allocation.