Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 2: Experiments and Analysis

TL;DR

This paper investigates the probability and existence of local minima in one-hidden-layer ReLU networks, showing that bad local minima are rare and gradient descent typically finds good solutions, supported by experiments on MNIST and CIFAR-10.

Contribution

It introduces a linear programming approach to identify genuine local minima and verifies that bad minima are almost nonexistent in practical scenarios.

Findings

Low probability of local minima in most regions for 1D Gaussian data

No bad differentiable local minima found almost everywhere in weight space

Gradient descent typically avoids trapping in poor local minima

Abstract

The existence of local minima for one-hidden-layer ReLU networks has been investigated theoretically in [8]. Based on the theory, in this paper, we first analyze how big the probability of existing local minima is for 1D Gaussian data and how it varies in the whole weight space. We show that this probability is very low in most regions. We then design and implement a linear programming based approach to judge the existence of genuine local minima, and use it to predict whether bad local minima exist for the MNIST and CIFAR-10 datasets, and find that there are no bad differentiable local minima almost everywhere in weight space once some hidden neurons are activated by samples. These theoretical predictions are verified experimentally by showing that gradient descent is not trapped in the cells from which it starts. We also perform experiments to explore the count and size of differentiable cells in the weight space.