On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting

TL;DR

This paper provides a theoretical analysis of training two-layer ReLU neural networks in a teacher-student setting, demonstrating conditions under which gradient descent can reliably recover the teacher network's parameters.

Contribution

It introduces a measure representation approach and a dual certificate argument to analyze global convergence and minima in training over-parameterized two-layer ReLU networks.

Findings

Gradient descent can identify teacher network parameters with high probability.

Regularization and over-parameterization are key for successful learning.

Theoretical tools include measure representation and dual certificates.

Abstract

Deep learning empirically achieves high performance in many applications, but its training dynamics has not been fully understood theoretically. In this paper, we explore theoretical analysis on training two-layer ReLU neural networks in a teacher-student regression model, in which a student network learns an unknown teacher network through its outputs. We show that with a specific regularization and sufficient over-parameterization, the student network can identify the parameters of the teacher network with high probability via gradient descent with a norm dependent stepsize even though the objective function is highly non-convex. The key theoretical tool is the measure representation of the neural networks and a novel application of a dual certificate argument for sparse estimation on a measure space. We analyze the global minima and global convergence property in the measure space.