Deep Manifold Part 1: Anatomy of Neural Network Manifold

TL;DR

This paper introduces a mathematical framework called Deep Manifold to analyze neural networks, revealing their infinite degrees of freedom, exponential capacity, and boundary conditions, while raising fundamental questions about training and convergence.

Contribution

It develops a novel neural network manifold framework based on numerical manifold principles, providing new insights into neural network structure, learning capacity, and convergence behavior.

Findings

Neural networks exhibit near infinite degrees of freedom.

Exponential learning capacity increases with depth.

Identifies key boundary conditions and bottlenecks in training.

Abstract

Based on the numerical manifold method principle, we developed a mathematical framework of a neural network manifold: Deep Manifold and discovered that neural networks: 1) is numerical computation combining forward and inverse; 2) have near infinite degrees of freedom; 3) exponential learning capacity with depth; 4) have self-progressing boundary conditions; 5) has training hidden bottleneck. We also define two concepts: neural network learning space and deep manifold space and introduce two concepts: neural network intrinsic pathway and fixed point. We raise three fundamental questions: 1). What is the training completion definition; 2). where is the deep learning convergence point (neural network fixed point); 3). How important is token timestamp in training data given negative time is critical in inverse problem.