Why should autoencoders work?

TL;DR

This paper investigates why autoencoders effectively reduce data to intrinsic low-dimensional structures, explaining their success through topological and differential topology principles, and providing theoretical guarantees for their performance.

Contribution

The paper offers a theoretical explanation for the effectiveness of autoencoders based on differential topology, highlighting topological obstructions and guarantees of near-perfect reconstruction.

Findings

Autoencoders can approximate homeomorphisms up to small errors.

Topological obstructions limit perfect reconstruction in theory.

Differential topology explains the practical success of autoencoders.

Abstract

Deep neural network autoencoders are routinely used computationally for model reduction. They allow recognizing the intrinsic dimension of data that lie in a $k$-dimensional subset $K$ of an input Euclidean space $\mathbb{R}^n$. The underlying idea is to obtain both an encoding layer that maps $\mathbb{R}^n$ into $\mathbb{R}^k$ (called the bottleneck layer or the space of latent variables) and a decoding layer that maps $\mathbb{R}^k$ back into $\mathbb{R}^n$, in such a way that the input data from the set $K$ is recovered when composing the two maps. This is achieved by adjusting parameters (weights) in the network to minimize the discrepancy between the input and the reconstructed output. Since neural networks (with continuous activation functions) compute continuous maps, the existence of a network that achieves perfect reconstruction would imply that $K$ is homeomorphic to a $k$-dimensional subset of $\mathbb{R}^k$, so clearly there are topological obstructions to finding such a network. On the other hand, in practice the technique is found to "work" well, which leads one to ask if there is a way to explain this effectiveness. We show that, up to small errors, indeed the method is guaranteed to work. This is done by appealing to certain facts from differential topology. A computational example is also included to illustrate the ideas.