Sparse Coding and Autoencoders

TL;DR

This paper rigorously analyzes whether gradient descent on a simple autoencoder can solve the dictionary learning problem, showing that under mild assumptions, the autoencoder's training landscape is favorable for recovering the underlying dictionary.

Contribution

It proves that gradient descent on an autoencoder's squared loss can effectively recover the dictionary in sparse coding, and demonstrates support recovery via ReLU layers independently of the loss.

Findings

Expected gradient norm is negligible near the true dictionary.

Experimental evidence supports local optimality of the true dictionary.

ReLU layers can recover sparse code support regardless of the loss function.

Abstract

In "Dictionary Learning" one tries to recover incoherent matrices $A^* \in \mathbb{R}^{n \times h}$ (typically overcomplete and whose columns are assumed to be normalized) and sparse vectors $x^* \in \mathbb{R}^h$ with a small support of size $h^p$ for some $0 <p < 1$ while having access to observations $y \in \mathbb{R}^n$ where $y = A^*x^*$. In this work we undertake a rigorous analysis of whether gradient descent on the squared loss of an autoencoder can solve the dictionary learning problem. The "Autoencoder" architecture we consider is a $\mathbb{R}^n \rightarrow \mathbb{R}^n$ mapping with a single ReLU activation layer of size $h$. Under very mild distributional assumptions on $x^*$, we prove that the norm of the expected gradient of the standard squared loss function is asymptotically (in sparse code dimension) negligible for all points in a small neighborhood of $A^*$. This is supported with experimental evidence using synthetic data. We also conduct experiments to suggest that $A^*$ is a local minimum. Along the way we prove that a layer of ReLU gates can be set up to automatically recover the support of the sparse codes. This property holds independent of the loss function. We believe that it could be of independent interest.