Joint-stochastic-approximation Autoencoders with Application to Semi-supervised Learning

TL;DR

This paper introduces JSA autoencoders, a new deep generative model framework that directly maximizes data likelihood and effectively handles discrete and continuous variables, improving semi-supervised learning performance.

Contribution

It presents the JSA autoencoder algorithm, the first successful application of discrete latent variables in semi-supervised learning, with theoretical backing and empirical validation.

Findings

JSA autoencoders outperform existing models in semi-supervised tasks.

They handle discrete and continuous variables robustly.

Discrete latent space models achieve competitive results on MNIST and SVHN.

Abstract

Our examination of existing deep generative models (DGMs), including VAEs and GANs, reveals two problems. First, their capability in handling discrete observations and latent codes is unsatisfactory, though there are interesting efforts. Second, both VAEs and GANs optimize some criteria that are indirectly related to the data likelihood. To address these problems, we formally present Joint-stochastic-approximation (JSA) autoencoders - a new family of algorithms for building deep directed generative models, with application to semi-supervised learning. The JSA learning algorithm directly maximizes the data log-likelihood and simultaneously minimizes the inclusive KL divergence the between the posteriori and the inference model. We provide theoretical results and conduct a series of experiments to show its superiority such as being robust to structure mismatch between encoder and decoder, consistent handling of both discrete and continuous variables. Particularly we empirically show that JSA autoencoders with discrete latent space achieve comparable performance to other state-of-the-art DGMs with continuous latent space in semi-supervised tasks over the widely adopted datasets - MNIST and SVHN. To the best of our knowledge, this is the first demonstration that discrete latent variable models are successfully applied in the challenging semi-supervised tasks.