Dive into Layers: Neural Network Capacity Bounding using Algebraic Geometry

TL;DR

This paper establishes a mathematical link between neural network size and learnability by using algebraic topology, specifically Betti numbers, to measure and bound the network's expressive capacity based on data complexity.

Contribution

It introduces a novel approach using Betti numbers from algebraic topology to analyze and bound neural network capacity, guiding architecture selection based on data complexity.

Findings

Betti numbers effectively measure input data complexity.

Neural network capacity is limited by layer scale.

Experimental validation on MNIST supports the theoretical analysis.

Abstract

The empirical results suggest that the learnability of a neural network is directly related to its size. To mathematically prove this, we borrow a tool in topological algebra: Betti numbers to measure the topological geometric complexity of input data and the neural network. By characterizing the expressive capacity of a neural network with its topological complexity, we conduct a thorough analysis and show that the network's expressive capacity is limited by the scale of its layers. Further, we derive the upper bounds of the Betti numbers on each layer within the network. As a result, the problem of architecture selection of a neural network is transformed to determining the scale of the network that can represent the input data complexity. With the presented results, the architecture selection of a fully connected network boils down to choosing a suitable size of the network such that it equips the Betti numbers that are not smaller than the Betti numbers of the input data. We perform the experiments on a real-world dataset MNIST and the results verify our analysis and conclusion. The code is publicly available.