Generative learning for deep networks

TL;DR

This paper introduces a probabilistic interpretation of deep neural networks using Bayesian inference, proposing a coupled recognition and generation model that can learn the full data distribution.

Contribution

It presents a novel approach linking DNNs to probabilistic models via Bayesian inference and suggests weight transposition for consistent recognition and generation networks.

Findings

Coupled recognition and generation networks can be trained to model full data distribution.

The proposed approach offers a unified probabilistic interpretation of DNNs.

Preliminary experiments show potential for effective generative and recognition tasks.

Abstract

Learning, taking into account full distribution of the data, referred to as generative, is not feasible with deep neural networks (DNNs) because they model only the conditional distribution of the outputs given the inputs. Current solutions are either based on joint probability models facing difficult estimation problems or learn two separate networks, mapping inputs to outputs (recognition) and vice-versa (generation). We propose an intermediate approach. First, we show that forward computation in DNNs with logistic sigmoid activations corresponds to a simplified approximate Bayesian inference in a directed probabilistic multi-layer model. This connection allows to interpret DNN as a probabilistic model of the output and all hidden units given the input. Second, we propose that in order for the recognition and generation networks to be more consistent with the joint model of the data, weights of the recognition and generator network should be related by transposition. We demonstrate in a tentative experiment that such a coupled pair can be learned generatively, modelling the full distribution of the data, and has enough capacity to perform well in both recognition and generation.