Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks

TL;DR

This paper analyzes the structure of critical points in two-layer neural networks, introducing new operators to understand their geometry and proving the existence of saddle branches, which aids in understanding network optimization.

Contribution

It introduces the critical embedding and reduction operators to analyze critical sets and proves saddle branch existence for certain network functions, advancing theoretical understanding.

Findings

Critical sets have a hierarchical structure.

Existence of saddle branches for certain critical points.

Provides foundational tools for neural network optimization study.

Abstract

This paper presents a comprehensive analysis of critical point sets in two-layer neural networks. To study such complex entities, we introduce the critical embedding operator and critical reduction operator as our tools. Given a critical point, we use these operators to uncover the whole underlying critical set representing the same output function, which exhibits a hierarchical structure. Furthermore, we prove existence of saddle branches for any critical set whose output function can be represented by a narrower network. Our results provide a solid foundation to the further study of optimization and training behavior of neural networks.