Deep Generative Stochastic Networks Trainable by Backprop

TL;DR

This paper introduces Generative Stochastic Networks (GSNs), a new probabilistic modeling framework that trains Markov chain transition operators using backpropagation, simplifying learning and enabling flexible sampling, including with missing data.

Contribution

The paper presents GSNs as an alternative to maximum likelihood, providing a theoretical foundation and practical training method that simplifies learning of complex generative models.

Findings

GSNs can be trained efficiently with backpropagation.

They perform well on image datasets, comparable to deep Boltzmann machines.

The framework allows sampling with missing inputs and subset variables.

Abstract

We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining.