On the Local Minima of the Empirical Risk

TL;DR

This paper introduces an algorithm for finding approximate local minima of the population risk in machine learning, despite only having access to a close approximation of the true risk, and establishes its optimality.

Contribution

The paper proposes a simple SGD-based algorithm for optimizing a smooth nonconvex function with approximate access, achieving near-optimal error tolerance bounds.

Findings

Algorithm guarantees to find approximate local minima within specified error bounds.

Provides lower bounds showing the optimality of the algorithm's error tolerance.

Applies results to sample complexity analysis for learning ReLU units.

Abstract

Population risk is always of primary interest in machine learning; however, learning algorithms only have access to the empirical risk. Even for applications with nonconvex nonsmooth losses (such as modern deep networks), the population risk is generally significantly more well-behaved from an optimization point of view than the empirical risk. In particular, sampling can create many spurious local minima. We consider a general framework which aims to optimize a smooth nonconvex function $F$ (population risk) given only access to an approximation $f$ (empirical risk) that is pointwise close to $F$ (i.e., $\|F-f\|_{\infty} \le \nu$). Our objective is to find the $\epsilon$-approximate local minima of the underlying function $F$ while avoiding the shallow local minima---arising because of the tolerance $\nu$---which exist only in $f$. We propose a simple algorithm based on stochastic gradient descent (SGD) on a smoothed version of $f$ that is guaranteed to achieve our goal as long as $\nu \le O(\epsilon^{1.5}/d)$. We also provide an almost matching lower bound showing that our algorithm achieves optimal error tolerance $\nu$ among all algorithms making a polynomial number of queries of $f$. As a concrete example, we show that our results can be directly used to give sample complexities for learning a ReLU unit.