Saturating Auto-Encoders

TL;DR

Saturating Auto-Encoders (SATAE) introduce a regularizer that encourages activations in saturated regions, limiting reconstruction of off-manifold inputs and enabling diverse feature learning with connections to existing auto-encoder variants.

Contribution

The paper proposes a novel saturation regularizer for auto-encoders with saturated activation functions, enhancing feature learning and establishing links to contractive and sparse auto-encoders.

Findings

SATAE limit reconstructions away from the data manifold.

Different activation functions lead to diverse learned features.

Connections to Contractive and Sparse Auto-Encoders are demonstrated.

Abstract

We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.