Switched linear encoding with rectified linear autoencoders

TL;DR

This paper investigates rectified linear autoencoders, revealing their connections to sparse coding models, providing intuitive interpretations, and demonstrating their behavior on artificial datasets.

Contribution

It offers a detailed analysis of rectified linear autoencoders and their relationship to sparse coding, enhancing understanding of their function and behavior.

Findings

Rectified linear autoencoders are closely related to sparse coding models.

The paper provides an intuitive interpretation of these models.

Demonstrations on artificial datasets validate the theoretical insights.

Abstract

Several recent results in machine learning have established formal connections between autoencoders---artificial neural network models that attempt to reproduce their inputs---and other coding models like sparse coding and K-means. This paper explores in depth an autoencoder model that is constructed using rectified linear activations on its hidden units. Our analysis builds on recent results to further unify the world of sparse linear coding models. We provide an intuitive interpretation of the behavior of these coding models and demonstrate this intuition using small, artificial datasets with known distributions.