A Theory of Generative ConvNet

TL;DR

This paper introduces a generative ConvNet model derived from discriminative ConvNets, which is piecewise Gaussian and uses an auto-encoder structure for image synthesis and learning.

Contribution

It establishes a novel connection between generative ConvNets and auto-encoders, providing a unique energy-based model with practical sampling and learning algorithms.

Findings

Model is piecewise Gaussian with auto-encoder basis functions

Langevin dynamics driven by auto-encoder reconstruction error

Maximum likelihood training synthesizes realistic images

Abstract

We show that a generative random field model, which we call generative ConvNet, can be derived from the commonly used discriminative ConvNet, by assuming a ConvNet for multi-category classification and assuming one of the categories is a base category generated by a reference distribution. If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process. The Langevin dynamics for sampling the generative ConvNet is driven by the reconstruction error of this auto-encoder. The contrastive divergence learning of the generative ConvNet reconstructs the training images by the auto-encoder. The maximum likelihood learning algorithm can synthesize realistic natural image patterns.