Knots in random neural networks

TL;DR

This paper investigates the properties of random neural networks, revealing that their number of knots closely matches the number of neurons, and models them as integrated random walks to explain this phenomenon.

Contribution

It provides an analytical framework linking random neural network behavior to integrated random walks, advancing understanding of their initial function complexity.

Findings

Number of knots approximately equals the number of neurons.

Random single-layer networks are equivalent to integrated random walks.

The analysis explains how network properties influence the number of knots.

Abstract

The weights of a neural network are typically initialized at random, and one can think of the functions produced by such a network as having been generated by a prior over some function space. Studying random networks, then, is useful for a Bayesian understanding of the network evolution in early stages of training. In particular, one can investigate why neural networks with huge numbers of parameters do not immediately overfit. We analyze the properties of random scalar-input feed-forward rectified linear unit architectures, which are random linear splines. With weights and biases sampled from certain common distributions, empirical tests show that the number of knots in the spline produced by the network is equal to the number of neurons, to very close approximation. We describe our progress towards a completely analytic explanation of this phenomenon. In particular, we show that random single-layer neural networks are equivalent to integrated random walks with variable step sizes. That each neuron produces one knot on average is equivalent to the associated integrated random walk having one zero crossing on average. We explore how properties of the integrated random walk, including the step sizes and initial conditions, affect the number of crossings. The number of knots in random neural networks can be related to the behavior of extreme learning machines, but it also establishes a prior preventing optimizers from immediately overfitting to noisy training data.