Stochastic Variance Reduction Gradient for a Non-convex Problem Using Graduated Optimization

TL;DR

This paper introduces two new algorithms, SVRG-GOA and PSVRG-GOA, that improve the efficiency of solving nonconvex optimization problems with convex and nonconvex parts using graduated optimization and variance reduction techniques.

Contribution

The paper proposes novel algorithms combining graduated optimization with SVRG for nonconvex problems, achieving lower iteration complexity and faster convergence.

Findings

Algorithms converge to global optima in nonconvex problems.

Proposed methods outperform GradOpt and nonconvex proximal SVRG in speed.

Theoretical analysis confirms improved iteration complexity.

Abstract

In machine learning, nonconvex optimization problems with multiple local optimums are often encountered. Graduated Optimization Algorithm (GOA) is a popular heuristic method to obtain global optimums of nonconvex problems through progressively minimizing a series of convex approximations to the nonconvex problems more and more accurate. Recently, such an algorithm GradOpt based on GOA is proposed with amazing theoretical and experimental results, but it mainly studies the problem which consists of one nonconvex part. This paper aims to find the global solution of a nonconvex objective with a convex part plus a nonconvex part based on GOA. By graduating approximating non-convex part of the problem and minimizing them with the Stochastic Variance Reduced Gradient (SVRG) or proximal SVRG, two new algorithms, SVRG-GOA and PSVRG-GOA, are proposed. We prove that the new algorithms have lower iteration complexity ($O(1/\varepsilon)$) than GradOpt ($O(1/\varepsilon^2)$). Some tricks, such as enlarging shrink factor, using project step, stochastic gradient, and mini-batch skills, are also given to accelerate the convergence speed of the proposed algorithms. Experimental results illustrate that the new algorithms with the similar performance can converge to 'global' optimums of the nonconvex problems, and they converge faster than the GradOpt and the nonconvex proximal SVRG.