Unraveling the Black Box of Neural Networks: A Dynamic Extremum Mapper

TL;DR

This paper challenges the black box perception of neural networks, showing their generalization relates to dynamic extrema mapping, and introduces a new linear-equation-based algorithm addressing issues like vanishing gradients.

Contribution

It provides a theoretical link between network parameters and extrema count, and proposes a novel algorithm distinct from back-propagation for training neural networks.

Findings

Number of extrema correlates with parameters

New algorithm solves parameters via linear equations

Framework explains gradient vanishing and overfitting

Abstract

We point out that neural networks are not black boxes, and their generalization stems from the ability to dynamically map a dataset to the extrema of the model function. We further prove that the number of extrema in a neural network is positively correlated with the number of its parameters. We then propose a new algorithm that is significantly different from back-propagation algorithm, which mainly obtains the values of parameters by solving a system of linear equations. Some difficult situations, such as gradient vanishing and overfitting, can be simply explained and dealt with in this framework.