Spurious Valleys in Two-layer Neural Network Optimization Landscapes

TL;DR

This paper investigates the loss surface topology of two-layer neural networks, showing that intrinsic dimension determines the presence of spurious valleys, which affects optimization success.

Contribution

It introduces the concept of intrinsic dimension and establishes necessary and sufficient conditions for the existence of spurious valleys in neural network loss landscapes.

Findings

Finite intrinsic dimension ensures no spurious valleys in overparametrized models.

Infinite intrinsic dimension can lead to spurious valleys for certain data distributions.

Spurious valleys are confined to low risk levels and are typically avoided in overparametrized models.

Abstract

Neural networks provide a rich class of high-dimensional, non-convex optimization problems. Despite their non-convexity, gradient-descent methods often successfully optimize these models. This has motivated a recent spur in research attempting to characterize properties of their loss surface that may explain such success. In this paper, we address this phenomenon by studying a key topological property of the loss: the presence or absence of spurious valleys, defined as connected components of sub-level sets that do not include a global minimum. Focusing on a class of two-layer neural networks defined by smooth (but generally non-linear) activation functions, we identify a notion of intrinsic dimension and show that it provides necessary and sufficient conditions for the absence of spurious valleys. More concretely, finite intrinsic dimension guarantees that for sufficiently overparametrised models no spurious valleys exist, independently of the data distribution. Conversely, infinite intrinsic dimension implies that spurious valleys do exist for certain data distributions, independently of model overparametrisation. Besides these positive and negative results, we show that, although spurious valleys may exist in general, they are confined to low risk levels and avoided with high probability on overparametrised models.