A Stochastic Gradient Method with Biased Estimation for Faster Nonconvex Optimization

TL;DR

This paper introduces a novel stochastic gradient method that uses biased estimation controlled by a hyper-parameter, enabling faster convergence in nonconvex optimization tasks like deep learning.

Contribution

It proposes an integrated approach that balances biased and unbiased gradient estimators via a hyper-parameter, improving convergence speed.

Findings

The method achieves faster convergence in nonconvex optimization.

The hyper-parameter effectively balances bias and variance.

Experimental results validate theoretical convergence improvements.

Abstract

A number of optimization approaches have been proposed for optimizing nonconvex objectives (e.g. deep learning models), such as batch gradient descent, stochastic gradient descent and stochastic variance reduced gradient descent. Theory shows these optimization methods can converge by using an unbiased gradient estimator. However, in practice biased gradient estimation can allow more efficient convergence to the vicinity since an unbiased approach is computationally more expensive. To produce fast convergence there are two trade-offs of these optimization strategies which are between stochastic/batch, and between biased/unbiased. This paper proposes an integrated approach which can control the nature of the stochastic element in the optimizer and can balance the trade-off of estimator between the biased and unbiased by using a hyper-parameter. It is shown theoretically and experimentally that this hyper-parameter can be configured to provide an effective balance to improve the convergence rate.