Adding One Neuron Can Eliminate All Bad Local Minima

TL;DR

This paper demonstrates that adding a single specially designed neuron with a skip connection to neural networks can eliminate all bad local minima, ensuring that every local minimum is globally optimal in binary classification tasks.

Contribution

The authors prove that a single neuron with a skip connection can transform the loss landscape so that all local minima are global, under mild assumptions.

Findings

Adding one neuron with skip connection guarantees global optimality at all local minima

The result applies to neural networks for binary classification

The landscape becomes more favorable for training algorithms

Abstract

One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.