Finite Sample Identification of Wide Shallow Neural Networks with Biases

TL;DR

This paper develops a new method with theoretical guarantees for identifying parameters of wide shallow neural networks with biases from finite samples, addressing a gap in existing literature.

Contribution

It introduces a constructive two-step approach for finite sample identification of wide shallow neural networks with biases, including theoretical analysis and empirical validation.

Findings

Effective parameter recovery demonstrated through numerical experiments.

Theoretical guarantees established for the proposed identification method.

Addresses a previously unresolved case of networks with biases in the finite sample setting.

Abstract

Artificial neural networks are functions depending on a finite number of parameters typically encoded as weights and biases. The identification of the parameters of the network from finite samples of input-output pairs is often referred to as the \emph{teacher-student model}, and this model has represented a popular framework for understanding training and generalization. Even if the problem is NP-complete in the worst case, a rapidly growing literature -- after adding suitable distributional assumptions -- has established finite sample identification of two-layer networks with a number of neurons $m=\mathcal O(D)$, $D$ being the input dimension. For the range $D<m<D^2$ the problem becomes harder, and truly little is known for networks parametrized by biases as well. This paper fills the gap by providing constructive methods and theoretical guarantees of finite sample identification for such wider shallow networks with biases. Our approach is based on a two-step pipeline: first, we recover the direction of the weights, by exploiting second order information; next, we identify the signs by suitable algebraic evaluations, and we recover the biases by empirical risk minimization via gradient descent. Numerical results demonstrate the effectiveness of our approach.