Fast Global Convergence via Landscape of Empirical Loss

TL;DR

This paper demonstrates that stochastic variance reduction methods can efficiently find the global optimum for certain non-convex loss functions in machine learning, leveraging the landscape of empirical loss.

Contribution

It proves linear convergence of stochastic variance reduction methods for non-convex M-estimators by exploiting the statistical properties of the population loss.

Findings

Stochastic variance reduction methods achieve global optimality with linear convergence.

Improved convergence analysis for batch gradient methods.

Insights into the landscape of empirical loss for non-convex optimization.

Abstract

While optimizing convex objective (loss) functions has been a powerhouse for machine learning for at least two decades, non-convex loss functions have attracted fast growing interests recently, due to many desirable properties such as superior robustness and classification accuracy, compared with their convex counterparts. The main obstacle for non-convex estimators is that it is in general intractable to find the optimal solution. In this paper, we study the computational issues for some non-convex M-estimators. In particular, we show that the stochastic variance reduction methods converge to the global optimal with linear rate, by exploiting the statistical property of the population loss. En route, we improve the convergence analysis for the batch gradient method in \cite{mei2016landscape}.