Towards Understanding Learning in Neural Networks with Linear Teachers

TL;DR

This paper proves that stochastic gradient descent can globally optimize a two-layer neural network with Leaky ReLU activations to learn linearly separable data, and explains why the resulting network often behaves approximately linearly.

Contribution

It provides the first theoretical proof of global optimization for this setting and links weight clustering to linear decision boundaries.

Findings

SGD globally optimizes the learning problem.

Networks often become approximately linear.

Weight clustering implies linear decision boundaries.

Abstract

Can a neural network minimizing cross-entropy learn linearly separable data? Despite progress in the theory of deep learning, this question remains unsolved. Here we prove that SGD globally optimizes this learning problem for a two-layer network with Leaky ReLU activations. The learned network can in principle be very complex. However, empirical evidence suggests that it often turns out to be approximately linear. We provide theoretical support for this phenomenon by proving that if network weights converge to two weight clusters, this will imply an approximately linear decision boundary. Finally, we show a condition on the optimization that leads to weight clustering. We provide empirical results that validate our theoretical analysis.