Autoencoders for a manifold learning problem with a Jacobian rank constraint

TL;DR

This paper introduces a novel autoencoder-based approach with Jacobian rank constraints to learn manifold structures, incorporating new regularization terms to improve the approximation of the underlying data manifold.

Contribution

It proposes a new method that enforces Jacobian rank constraints in autoencoders, enhancing their ability to learn low-dimensional manifolds beyond architectural limitations.

Findings

The method effectively reduces the autoencoder's output dimension.

The approach outperforms existing methods on synthetic datasets.

In experiments, the new regularization improves manifold learning accuracy.

Abstract

We formulate the manifold learning problem as the problem of finding an operator that maps any point to a close neighbor that lies on a ``hidden'' $k$-dimensional manifold. We call this operator the correcting function. Under this formulation, autoencoders can be viewed as a tool to approximate the correcting function. Given an autoencoder whose Jacobian has rank $k$, we deduce from the classical Constant Rank Theorem that its range has a structure of a $k$-dimensional manifold. A $k$-dimensionality of the range can be forced by the architecture of an autoencoder (by fixing the dimension of the code space), or alternatively, by an additional constraint that the rank of the autoencoder mapping is not greater than $k$. This constraint is included in the objective function as a new term, namely a squared Ky-Fan $k$-antinorm of the Jacobian function. We claim that this constraint is a factor that effectively reduces the dimension of the range of an autoencoder, additionally to the reduction defined by the architecture. We also add a new curvature term into the objective. To conclude, we experimentally compare our approach with the CAE+H method on synthetic and real-world datasets.