Some Insights into the Geometry and Training of Neural Networks

TL;DR

This paper offers new insights into neural network training and classification by analyzing decision regions, weight-bias relationships, and their impact on gradients, regularization, and data sampling.

Contribution

It provides a feature-space perspective on neural networks, linking weight parameters and decision boundaries to training dynamics and generalization.

Findings

Decision regions are influenced by weight and bias configurations.

Gradient backpropagation is affected by the structure of decision regions.

Scaling weights relates to training sample density and impacts vanishing gradients.

Abstract

Neural networks have been successfully used for classification tasks in a rapidly growing number of practical applications. Despite their popularity and widespread use, there are still many aspects of training and classification that are not well understood. In this paper we aim to provide some new insights into training and classification by analyzing neural networks from a feature-space perspective. We review and explain the formation of decision regions and study some of their combinatorial aspects. We place a particular emphasis on the connections between the neural network weight and bias terms and properties of decision boundaries and other regions that exhibit varying levels of classification confidence. We show how the error backpropagates in these regions and emphasize the important role they have in the formation of gradients. These findings expose the connections between scaling of the weight parameters and the density of the training samples. This sheds more light on the vanishing gradient problem, explains the need for regularization, and suggests an approach for subsampling training data to improve performance.