Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networks

TL;DR

This paper establishes conditions under which leaky ReLU two-layer neural networks trained with gradient descent exhibit benign overfitting on mixture data, extending previous results by proving directional convergence and identifying phase transitions.

Contribution

It introduces the first proof of directional convergence for leaky ReLU networks trained with gradient descent on mixture data, broadening understanding of benign overfitting beyond orthogonal data.

Findings

Benign overfitting occurs in a wider range of scenarios than previously known.

Directional convergence is established for leaky ReLU networks trained with gradient descent.

A new phase transition in classification error is identified.

Abstract

In this paper, we provide sufficient conditions of benign overfitting of fixed width leaky ReLU two-layer neural network classifiers trained on mixture data via gradient descent. Our results are derived by establishing directional convergence of the network parameters and classification error bound of the convergent direction. Our classification error bound also lead to the discovery of a newly identified phase transition. Previously, directional convergence in (leaky) ReLU neural networks was established only for gradient flow. Due to the lack of directional convergence, previous results on benign overfitting were limited to those trained on nearly orthogonal data. All of our results hold on mixture data, which is a broader data setting than the nearly orthogonal data setting in prior work. We demonstrate our findings by showing that benign overfitting occurs with high probability in a much wider range of scenarios than previously known. Our results also allow us to characterize cases when benign overfitting provably fails even if directional convergence occurs. Our work thus provides a more complete picture of benign overfitting in leaky ReLU two-layer neural networks.