A backward pass through a CNN using a generative model of its activations

TL;DR

This paper introduces a method for performing a backward pass through a CNN by creating a generative model of its activations, enabling diverse queries like object reconstruction and handling occlusion.

Contribution

It proposes a novel generative model of CNN activations that facilitates a backward inference pass, unifying top-down and bottom-up information processing.

Findings

Enables object instance detection and reconstruction from CNN activations

Addresses occlusion and ambiguous inputs with a backward inference method

Provides a unified framework for various CNN query types

Abstract

Neural networks have shown to be a practical way of building a very complex mapping between a pre-specified input space and output space. For example, a convolutional neural network (CNN) mapping an image into one of a thousand object labels is approaching human performance in this particular task. However the mapping (neural network) does not automatically lend itself to other forms of queries, for example, to detect/reconstruct object instances, to enforce top-down signal on ambiguous inputs, or to recover object instances from occlusion. One way to address these queries is a backward pass through the network that fuses top-down and bottom-up information. In this paper, we show a way of building such a backward pass by defining a generative model of the neural network's activations. Approximate inference of the model would naturally take the form of a backward pass through the CNN layers, and it addresses the aforementioned queries in a unified framework.