Discriminative Recurrent Sparse Auto-Encoders

TL;DR

This paper introduces a discriminative recurrent sparse auto-encoder that combines unsupervised and supervised learning in a deep, recurrent architecture, effectively learning hierarchical representations and achieving high performance on MNIST with fewer parameters.

Contribution

It proposes a novel recurrent auto-encoder model that integrates discriminative training, hierarchical organization, and reduced parameter count, advancing deep unsupervised and supervised learning methods.

Findings

Units differentiate into class-specific prototypes and deformation parts.

Hierarchical organization with part-units driven by input and categorical-units building over time.

Achieves excellent MNIST performance with few hidden units.

Abstract

We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss function is then augmented with a discriminative term on the supervised classification. The depth implicit in the temporally-unrolled form allows the system to exhibit all the power of deep networks, while substantially reducing the number of trainable parameters. From an initially unstructured network the hidden units differentiate into categorical-units, each of which represents an input prototype with a well-defined class; and part-units representing deformations of these prototypes. The learned organization of the recurrent encoder is hierarchical: part-units are driven directly by the input, whereas the activity of categorical-units builds up over time through interactions with the part-units. Even using a small number of hidden units per layer, discriminative recurrent sparse auto-encoders achieve excellent performance on MNIST.