Benignity of loss landscape with weight decay requires both large overparametrization and initialization

TL;DR

This paper demonstrates that for two-layer ReLU networks with weight decay, a benign loss landscape free of spurious minima occurs only under large overparameterization and initialization, highlighting the importance of these factors.

Contribution

It provides a theoretical analysis showing the conditions under which the loss landscape becomes benign in regularized neural networks, emphasizing the roles of overparameterization and initialization.

Findings

Benign landscape occurs with large overparametrization when m ≥ min(n^d, 2^n).

Almost all constant activation regions contain a global minimum under these conditions.

Small initializations can still lead to spurious minima despite benign landscapes at large initializations.

Abstract

The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint. While weight decay is standard practice in modern training procedures, most theoretical analyses focus on unregularized settings. In this work, we investigate the loss landscape of the $\ell_2$-regularized training loss for two-layer ReLU networks. We show that the landscape becomes benign -- i.e., free of spurious local minima -- under large overparametrization, specifically when the network width $m$ satisfies $m \gtrsim \min(n^d, 2^n)$, where $n$ is the number of data points and $d$ the input dimension. More precisely in this regime, almost all constant activation regions contain a global minimum and no spurious local minima. We further show that this level of overparametrization is not only sufficient but also necessary via the example of orthogonal data. Finally, we demonstrate that such loss landscape results primarily hold relevance in the large initialization regime. In contrast, for small initializations -- corresponding to the feature learning regime -- optimization can still converge to spurious local minima, despite the global benignity of the landscape.