On the Benefit of Width for Neural Networks: Disappearance of Bad Basins

TL;DR

This paper proves a phase transition in neural network loss landscapes, showing that increasing width eliminates sub-optimal basins, thereby explaining the optimization benefits of wide networks.

Contribution

It rigorously demonstrates the existence of a phase transition in the loss surface of neural networks as width increases, from having sub-optimal basins to none.

Findings

Wide networks have no sub-optimal basins in their loss surface.

Narrow networks can have strict local minima that are not global.

A phase transition occurs at a certain width threshold.

Abstract

Wide networks are often believed to have a nice optimization landscape, but what rigorous results can we prove? To understand the benefit of width, it is important to identify the difference between wide and narrow networks. In this work, we prove that from narrow to wide networks, there is a phase transition from having sub-optimal basins to no sub-optimal basins. Specifically, we prove two results: on the positive side, for any continuous activation functions, the loss surface of a class of wide networks has no sub-optimal basins, where "basin" is defined as the set-wise strict local minimum; on the negative side, for a large class of networks with width below a threshold, we construct strict local minima that are not global. These two results together show the phase transition from narrow to wide networks.