On Graduated Optimization for Stochastic Non-Convex Problems

TL;DR

This paper introduces a new first-order graduated optimization algorithm with proven convergence guarantees for certain non-convex functions, extending to stochastic and zero-order settings with specific convergence rates.

Contribution

The paper provides the first theoretical convergence analysis of a graduated optimization algorithm for non-convex problems, including stochastic and zero-order variants.

Findings

Converges to an ε-approximate solution within O(1/ε²) steps for certain non-convex functions.

Extends to stochastic optimization with the same convergence rate.

Zero-order variant converges at rate O(d²/ε⁴).

Abstract

The graduated optimization approach, also known as the continuation method, is a popular heuristic to solving non-convex problems that has received renewed interest over the last decade. Despite its popularity, very little is known in terms of theoretical convergence analysis. In this paper we describe a new first-order algorithm based on graduated optimiza- tion and analyze its performance. We characterize a parameterized family of non- convex functions for which this algorithm provably converges to a global optimum. In particular, we prove that the algorithm converges to an {\epsilon}-approximate solution within O(1/\epsilon^2) gradient-based steps. We extend our algorithm and analysis to the setting of stochastic non-convex optimization with noisy gradient feedback, attaining the same convergence rate. Additionally, we discuss the setting of zero-order optimization, and devise a a variant of our algorithm which converges at rate of O(d^2/\epsilon^4).