Slope and generalization properties of neural networks

TL;DR

This paper introduces the concept of controlling the slope of neural networks to improve generalization, providing theoretical insights and empirical evidence that slope distribution is architecture-independent and smoothly varying.

Contribution

It proposes the slope as a measure to control neural network complexity, with theoretical properties and empirical validation across different architectures.

Findings

Slope distribution is independent of layer width in trained networks.

Mean slope has weak dependence on architecture.

Slope varies smoothly and aligns with theoretical predictions.

Abstract

Neural networks are very successful tools in for example advanced classification. From a statistical point of view, fitting a neural network may be seen as a kind of regression, where we seek a function from the input space to a space of classification probabilities that follows the "general" shape of the data, but avoids overfitting by avoiding memorization of individual data points. In statistics, this can be done by controlling the geometric complexity of the regression function. We propose to do something similar when fitting neural networks by controlling the slope of the network. After defining the slope and discussing some of its theoretical properties, we go on to show empirically in examples, using ReLU networks, that the distribution of the slope of a well-trained neural network classifier is generally independent of the width of the layers in a fully connected network, and that the mean of the distribution only has a weak dependence on the model architecture in general. The slope is of similar size throughout the relevant volume, and varies smoothly. It also behaves as predicted in rescaling examples. We discuss possible applications of the slope concept, such as using it as a part of the loss function or stopping criterion during network training, or ranking data sets in terms of their complexity.