Efficient Representation of Low-Dimensional Manifolds using Deep Networks

TL;DR

This paper demonstrates that deep neural networks can efficiently learn and represent data lying on low-dimensional manifolds within high-dimensional spaces, capturing intrinsic structures with minimal parameters.

Contribution

It introduces methods showing deep networks can exactly embed monotonic chain manifolds and extend to general manifolds, using near-optimal parameters.

Findings

Deep networks can exactly embed points on monotonic chain manifolds.

Networks project nearby points onto the manifold with minimal error.

Results extend to more general low-dimensional manifolds.

Abstract

We consider the ability of deep neural networks to represent data that lies near a low-dimensional manifold in a high-dimensional space. We show that deep networks can efficiently extract the intrinsic, low-dimensional coordinates of such data. We first show that the first two layers of a deep network can exactly embed points lying on a monotonic chain, a special type of piecewise linear manifold, mapping them to a low-dimensional Euclidean space. Remarkably, the network can do this using an almost optimal number of parameters. We also show that this network projects nearby points onto the manifold and then embeds them with little error. We then extend these results to more general manifolds.