Convergence Analysis of the Dynamics of a Special Kind of Two-Layered Neural Networks with $\ell_1$ and $\ell_2$ Regularization

TL;DR

This paper extends convergence analysis for two-layer ReLU neural networks by incorporating $$ and $$ regularization, proving convergence to optimal solutions under certain conditions, supported by numerical experiments.

Contribution

It provides a theoretical convergence guarantee for regularized two-layer neural networks with ReLU activation, considering both  and  regularization terms.

Findings

Weight vectors converge to the optimal solution with high probability.

Small regularization coefficient  or  ensures convergence.

Numerical experiments validate the theoretical results.

Abstract

In this paper, we made an extension to the convergence analysis of the dynamics of two-layered bias-free networks with one $ReLU$ output. We took into consideration two popular regularization terms: the $\ell_1$ and $\ell_2$ norm of the parameter vector $w$, and added it to the square loss function with coefficient $\lambda/2$. We proved that when $\lambda$ is small, the weight vector $w$ converges to the optimal solution $\hat{w}$ (with respect to the new loss function) with probability $\geq (1-\varepsilon)(1-A_d)/2$ under random initiations in a sphere centered at the origin, where $\varepsilon$ is a small value and $A_d$ is a constant. Numerical experiments including phase diagrams and repeated simulations verified our theory.