How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

TL;DR

This paper investigates how ReLU activation influences the implicit bias of gradient descent in high-dimensional neural network regression, revealing that the bias approximates the minimum-l2-norm solution with a quantifiable gap.

Contribution

It provides a novel primal-dual analysis characterizing the implicit bias of GD for shallow ReLU models on high-dimensional data, bridging previous theoretical gaps.

Findings

Implicit bias approximates minimum-l2-norm solution

Gap on the order of sqrt(n/d) between bias and minimum-l2-norm

ReLU activation pattern stabilizes quickly with high probability

Abstract

Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work showed that the implicit bias does not exist in the worst-case (Vardi and Shamir, 2021), or corresponds exactly to the minimum-l2-norm solution among all global minima under exactly orthogonal data (Boursier et al., 2022). Our work interpolates between these two extremes and shows that, for sufficiently high-dimensional random data, the implicit bias approximates the minimum-l2-norm solution with high probability with a gap on the order $\Theta(\sqrt{n/d})$, where n is the number of training examples and d is the feature dimension. Our results are obtained through a novel primal-dual analysis, which carefully tracks the evolution of predictions, data-span coefficients, as well as their interactions, and shows that the ReLU activation pattern quickly stabilizes with high probability over the random data.