Covariate-informed Representation Learning to Prevent Posterior Collapse of iVAE

TL;DR

This paper introduces CI-iVAE, a novel covariate-informed variational autoencoder that addresses the posterior collapse issue in iVAE, leading to richer latent representations and improved performance across diverse datasets.

Contribution

The paper proposes a new objective function for iVAE that prevents posterior collapse by incorporating a mixture of encoder and posterior distributions, enhancing latent information.

Findings

Prevents posterior collapse in iVAE.

Achieves tighter evidence lower bounds.

Improves latent representation quality on multiple datasets.

Abstract

The recently proposed identifiable variational autoencoder (iVAE) framework provides a promising approach for learning latent independent components (ICs). iVAEs use auxiliary covariates to build an identifiable generation structure from covariates to ICs to observations, and the posterior network approximates ICs given observations and covariates. Though the identifiability is appealing, we show that iVAEs could have local minimum solution where observations and the approximated ICs are independent given covariates.-a phenomenon we referred to as the posterior collapse problem of iVAEs. To overcome this problem, we develop a new approach, covariate-informed iVAE (CI-iVAE) by considering a mixture of encoder and posterior distributions in the objective function. In doing so, the objective function prevents the posterior collapse, resulting latent representations that contain more information of the observations. Furthermore, CI-iVAEs extend the original iVAE objective function to a larger class and finds the optimal one among them, thus having tighter evidence lower bounds than the original iVAE. Experiments on simulation datasets, EMNIST, Fashion-MNIST, and a large-scale brain imaging dataset demonstrate the effectiveness of our new method.