The layer-wise L1 Loss Landscape of Neural Nets is more complex around local minima

TL;DR

This paper investigates the complex structure of the L1 loss landscape of neural networks around local minima, revealing slow decay and increased vertex density that could inform improved gradient descent algorithms.

Contribution

It introduces an analysis of the L1 loss landscape around local minima using the Deep ReLU Simplex algorithm, highlighting new insights into loss decay and vertex density behavior.

Findings

Loss decays exponentially slow near local minima

Vertex density increases significantly around minima

Loss level can be estimated before convergence

Abstract

For fixed training data and network parameters in the other layers the L1 loss of a ReLU neural network as a function of the first layer's parameters is a piece-wise affine function. We use the Deep ReLU Simplex algorithm to iteratively minimize the loss monotonically on adjacent vertices and analyze the trajectory of these vertex positions. We empirically observe that in a neighbourhood around a local minimum, the iterations behave differently such that conclusions on loss level and proximity of the local minimum can be made before it has been found: Firstly the loss seems to decay exponentially slow at iterated adjacent vertices such that the loss level at the local minimum can be estimated from the loss levels of subsequently iterated vertices, and secondly we observe a strong increase of the vertex density around local minima. This could have far-reaching consequences for the design of new gradient-descent algorithms that might improve convergence rate by exploiting these facts.