A Variance Controlled Stochastic Method with Biased Estimation for Faster Non-convex Optimization

TL;DR

This paper introduces VCSG, a variance-controlled stochastic gradient method that balances biased and unbiased estimations, reducing gradient calculations and improving convergence speed for non-convex optimization.

Contribution

The paper proposes VCSG, a novel stochastic gradient method with a variance control parameter and batch variance reduction, enhancing convergence efficiency for non-convex functions.

Findings

VCSG converges to an approximate stationary point within optimal gradient complexity.

The method reduces the number of full gradient computations needed.

Experimental results demonstrate improved convergence over existing methods.

Abstract

In this paper, we proposed a new technique, {\em variance controlled stochastic gradient} (VCSG), to improve the performance of the stochastic variance reduced gradient (SVRG) algorithm. To avoid over-reducing the variance of gradient by SVRG, a hyper-parameter $\lambda$ is introduced in VCSG that is able to control the reduced variance of SVRG. Theory shows that the optimization method can converge by using an unbiased gradient estimator, but in practice, biased gradient estimation can allow more efficient convergence to the vicinity since an unbiased approach is computationally more expensive. $\lambda$ also has the effect of balancing the trade-off between unbiased and biased estimations. Secondly, to minimize the number of full gradient calculations in SVRG, a variance-bounded batch is introduced to reduce the number of gradient calculations required in each iteration. For smooth non-convex functions, the proposed algorithm converges to an approximate first-order stationary point (i.e. $\mathbb{E}\|\nabla{f}(x)\|^{2}\leq\epsilon$) within $\mathcal{O}(min\{1/\epsilon^{3/2},n^{1/4}/\epsilon\})$ number of stochastic gradient evaluations, which improves the leading gradient complexity of stochastic gradient-based method SCS $(\mathcal{O}(min\{1/\epsilon^{5/3},n^{2/3}/\epsilon\})$. It is shown theoretically and experimentally that VCSG can be deployed to improve convergence.