Encoding Binary Concepts in the Latent Space of Generative Models for Enhancing Data Representation

TL;DR

This paper introduces a binarized regularization method for autoencoders that enhances the learning of binary concepts, leading to better disentanglement, improved data generation, and alleviation of catastrophic forgetting in continual learning.

Contribution

It proposes a novel binarizing hyperparameter and regularization technique that can be integrated into existing VAE variants to improve their disentanglement, reconstruction, and transferability.

Findings

Improves disentanglement of latent space.

Enhances data generation quality.

Reduces catastrophic forgetting in continual learning.

Abstract

Binary concepts are empirically used by humans to generalize efficiently. And they are based on Bernoulli distribution which is the building block of information. These concepts span both low-level and high-level features such as "large vs small" and "a neuron is active or inactive". Binary concepts are ubiquitous features and can be used to transfer knowledge to improve model generalization. We propose a novel binarized regularization to facilitate learning of binary concepts to improve the quality of data generation in autoencoders. We introduce a binarizing hyperparameter $r$ in data generation process to disentangle the latent space symmetrically. We demonstrate that this method can be applied easily to existing variational autoencoder (VAE) variants to encourage symmetric disentanglement, improve reconstruction quality, and prevent posterior collapse without computation overhead. We also demonstrate that this method can boost existing models to learn more transferable representations and generate more representative samples for the input distribution which can alleviate catastrophic forgetting using generative replay under continual learning settings.