Saddlepoints in Unsupervised Least Squares

TL;DR

This paper analyzes the risk landscape of unsupervised least squares in deep auto-encoders, revealing that all non-trivial critical points are saddlepoints and proposing a new regularization-based optimization strategy.

Contribution

It establishes an equivalence between unsupervised least squares and principal manifolds, providing new insights into the saddlepoint structure of auto-encoder risk landscapes.

Findings

All non-trivial critical points are saddlepoints.

Overcomplete auto-encoders have degenerate saddlepoints.

Proposes a new regularization-based optimization strategy.

Abstract

This paper sheds light on the risk landscape of unsupervised least squares in the context of deep auto-encoding neural nets. We formally establish an equivalence between unsupervised least squares and principal manifolds. This link provides insight into the risk landscape of auto--encoding under the mean squared error, in particular all non-trivial critical points are saddlepoints. Finding saddlepoints is in itself difficult, overcomplete auto-encoding poses the additional challenge that the saddlepoints are degenerate. Within this context we discuss regularization of auto-encoders, in particular bottleneck, denoising and contraction auto-encoding and propose a new optimization strategy that can be framed as particular form of contractive regularization.