A Theoretical Study of Neural Network Expressive Power via Manifold Topology

TL;DR

This paper explores how the topological complexity of data manifolds influences the size requirements of neural networks, providing a combined topological and geometric analysis to establish size upper bounds.

Contribution

It introduces a novel theoretical framework that incorporates manifold topology into the analysis of neural network expressive power, extending beyond geometric considerations.

Findings

Derived an upper bound on neural network size based on manifold topology.

Integrated topological and geometric properties to analyze data manifold complexity.

Highlight the importance of topology in neural network capacity analysis.

Abstract

A prevalent assumption regarding real-world data is that it lies on or close to a low-dimensional manifold. When deploying a neural network on data manifolds, the required size, i.e., the number of neurons of the network, heavily depends on the intricacy of the underlying latent manifold. While significant advancements have been made in understanding the geometric attributes of manifolds, it's essential to recognize that topology, too, is a fundamental characteristic of manifolds. In this study, we investigate network expressive power in terms of the latent data manifold. Integrating both topological and geometric facets of the data manifold, we present a size upper bound of ReLU neural networks.