Agnostic Learning of General ReLU Activation Using Gradient Descent

TL;DR

This paper analyzes the convergence of gradient descent for agnostically learning ReLU functions with non-zero bias under Gaussian distributions, providing guarantees on the learned function's error relative to the best possible ReLU.

Contribution

It extends previous analyses to include non-zero bias ReLUs, offering finite sample guarantees and broader distribution applicability.

Findings

Gradient descent converges to a near-optimal ReLU with high probability.

The analysis applies to a broader class of distributions beyond Gaussians.

Finite sample guarantees are established for the learning process.

Abstract

We provide a convergence analysis of gradient descent for the problem of agnostically learning a single ReLU function with moderate bias under Gaussian distributions. Unlike prior work that studies the setting of zero bias, we consider the more challenging scenario when the bias of the ReLU function is non-zero. Our main result establishes that starting from random initialization, in a polynomial number of iterations gradient descent outputs, with high probability, a ReLU function that achieves an error that is within a constant factor of the optimal error of the best ReLU function with moderate bias. We also provide finite sample guarantees, and these techniques generalize to a broader class of marginal distributions beyond Gaussians.