Cauchy-Schwarz Regularized Autoencoder

TL;DR

This paper introduces a novel autoencoder framework with a Cauchy-Schwarz divergence-based objective, enabling the use of Gaussian mixture priors for improved density estimation, clustering, and semi-supervised learning.

Contribution

It develops an analytical solution for GMM priors in autoencoders using a Cauchy-Schwarz divergence, enhancing the model's ability to represent complex data distributions.

Findings

Improved density estimation over standard VAEs.

Enhanced clustering and semi-supervised learning performance.

Better face analysis results with richer priors.

Abstract

Recent work in unsupervised learning has focused on efficient inference and learning in latent variables models. Training these models by maximizing the evidence (marginal likelihood) is typically intractable. Thus, a common approximation is to maximize the Evidence Lower BOund (ELBO) instead. Variational autoencoders (VAE) are a powerful and widely-used class of generative models that optimize the ELBO efficiently for large datasets. However, the VAE's default Gaussian choice for the prior imposes a strong constraint on its ability to represent the true posterior, thereby degrading overall performance. A Gaussian mixture model (GMM) would be a richer prior, but cannot be handled efficiently within the VAE framework because of the intractability of the Kullback-Leibler divergence for GMMs. We deviate from the common VAE framework in favor of one with an analytical solution for Gaussian mixture prior. To perform efficient inference for GMM priors, we introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. This new objective allows us to incorporate richer, multi-modal priors into the autoencoding framework. We provide empirical studies on a range of datasets and show that our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.