A Stochastic Gradient Descent Method for Globally Minimizing Nearly Convex Functions

TL;DR

This paper introduces a stochastic gradient descent method with adaptive noise for globally minimizing nearly convex functions, achieving linear convergence under certain conditions and demonstrating effectiveness through numerical experiments.

Contribution

The paper presents a novel SGD algorithm with adaptive Gaussian noise for global minimization of nearly convex functions, including a double-loop scheme for improved convergence.

Findings

Converges linearly to a neighborhood of the global minimum.

Neighborhood size depends on gradient variance and lower bound accuracy.

Numerical experiments confirm the method's effectiveness.

Abstract

This paper proposes a stochastic gradient descent method with an adaptive Gaussian noise term for the global minimization of nearly convex functions, which are nonconvex and possess multiple strict local minimizers. The noise term, independent of the gradient, is determined by the difference between the current function value and a lower bound estimate of the optimal value. In both probability space and state space, we show that the proposed algorithm converges linearly to a neighborhood of the global optimal solution. The size of this neighborhood depends on the variance of the gradient and the deviation between the estimated lower bound and the optimal value. In particular, when full gradient information is available and a sharp lower bound of the objective function is provided, the algorithm achieves linear convergence to the global optimum. Furthermore, we introduce a double-loop scheme that alternately updates the lower bound estimate and the optimization sequence, enabling convergence to a neighborhood of the global optimum that depends solely on the gradient variance. Numerical experiments on several benchmark problems demonstrate the effectiveness of the proposed algorithm.