Cascading Denoising Auto-Encoder as a Deep Directed Generative Model

TL;DR

This paper introduces Cascading Denoising Auto-Encoders (CDAE), a deep directed generative model that effectively generates data samples without intractable inference, addressing limitations of previous denoising auto-encoder approaches.

Contribution

It proposes a novel cascading architecture combining Denoising Auto-Encoders with stochastic identity mappings to enable tractable sampling and training of deep generative models on complex datasets.

Findings

CDAE can generate data samples from complex distributions.

The model trains successfully without intractable posterior inference.

It provides a simple framework for modeling complex datasets.

Abstract

Recent work (Bengio et al., 2013) has shown howDenoising Auto-Encoders(DAE) become gener-ative models as a density estimator. However,in practice, the framework suffers from a mixingproblem in the MCMC sampling process and nodirect method to estimate the test log-likelihood.We consider a directed model with an stochas-tic identity mapping (simple corruption pro-cess) as an inference model and a DAE as agenerative model. By cascading these mod-els, we propose Cascading Denoising Auto-Encoders(CDAE) which can generate samples ofdata distribution from tractable prior distributionunder the assumption that probabilistic distribu-tion of corrupted data approaches tractable priordistribution as the level of corruption increases.This work tries to answer two questions. On theone hand, can deep directed models be success-fully trained without intractable posterior infer-ence and difficult optimization of very deep neu-ral networks in inference and generative mod-els? These are unavoidable when recent suc-cessful directed model like VAE (Kingma &Welling, 2014) is trained on complex dataset likereal images. On the other hand, can DAEs getclean samples of data distribution from heavilycorrupted samples which can be considered oftractable prior distribution far from data mani-fold? so-called global denoising scheme.Our results show positive responses of thesequestions and this work can provide fairly simpleframework for generative models of very com-plex dataset.