How regularization affects the geometry of loss functions

TL;DR

This paper investigates how various regularizers influence the geometric properties of neural network loss functions, particularly focusing on whether regularization can make the loss function Morse, which impacts the learning dynamics.

Contribution

It provides a theoretical analysis of the effects of different regularizers on the Morse property of neural network loss functions, highlighting conditions under which regularization induces Morse characteristics.

Findings

Regularizers can transform non-Morse loss functions into Morse functions.

The study identifies specific regularizers that promote Morse properties in deep neural networks.

Implications for understanding the geometry of loss landscapes and optimization dynamics.

Abstract

What neural networks learn depends fundamentally on the geometry of the underlying loss function. We study how different regularizers affect the geometry of this function. One of the most basic geometric properties of a smooth function is whether it is Morse or not. For nonlinear deep neural networks, the unregularized loss function $L$ is typically not Morse. We consider several different regularizers, including weight decay, and study for which regularizers the regularized function $L_\epsilon$ becomes Morse.