Learning and generalization of one-hidden-layer neural networks, going beyond standard Gaussian data

TL;DR

This paper studies the training and generalization of one-hidden-layer neural networks with Gaussian mixture inputs, providing theoretical guarantees on convergence, sample complexity, and the influence of input distributions.

Contribution

It offers the first theoretical analysis of how Gaussian mixture input distributions affect learning rates and sample complexity in neural networks.

Findings

Linear convergence to critical points with finite samples

Characterization of input distribution impact on sample complexity

Guaranteed generalization error bounds

Abstract

This paper analyzes the convergence and generalization of training a one-hidden-layer neural network when the input features follow the Gaussian mixture model consisting of a finite number of Gaussian distributions. Assuming the labels are generated from a teacher model with an unknown ground truth weight, the learning problem is to estimate the underlying teacher model by minimizing a non-convex risk function over a student neural network. With a finite number of training samples, referred to the sample complexity, the iterations are proved to converge linearly to a critical point with guaranteed generalization error. In addition, for the first time, this paper characterizes the impact of the input distributions on the sample complexity and the learning rate.