Blocked and Hierarchical Disentangled Representation From Information Theory Perspective

TL;DR

This paper introduces BHiVAE, a novel hierarchical variational autoencoder based on information theory, designed to improve disentangled representations through block structures and hierarchical organization, with both supervised and unsupervised variants.

Contribution

It presents a new theoretical model, BHiVAE, that leverages information bottleneck and maximization principles to enhance disentanglement via hierarchical blocks and a novel prior, applicable in supervised and unsupervised settings.

Findings

Achieves excellent disentanglement in experiments.

Demonstrates superior classification accuracy.

Effectively separates attributes using hierarchical blocks.

Abstract

We propose a novel and theoretical model, blocked and hierarchical variational autoencoder (BHiVAE), to get better-disentangled representation. It is well known that information theory has an excellent explanatory meaning for the network, so we start to solve the disentanglement problem from the perspective of information theory. BHiVAE mainly comes from the information bottleneck theory and information maximization principle. Our main idea is that (1) Neurons block not only one neuron node is used to represent attribute, which can contain enough information; (2) Create a hierarchical structure with different attributes on different layers, so that we can segment the information within each layer to ensure that the final representation is disentangled. Furthermore, we present supervised and unsupervised BHiVAE, respectively, where the difference is mainly reflected in the separation of information between different blocks. In supervised BHiVAE, we utilize the label information as the standard to separate blocks. In unsupervised BHiVAE, without extra information, we use the Total Correlation (TC) measure to achieve independence, and we design a new prior distribution of the latent space to guide the representation learning. It also exhibits excellent disentanglement results in experiments and superior classification accuracy in representation learning.