Stopping Rules for Gradient Methods for Non-Convex Problems with Additive Noise in Gradient

TL;DR

This paper develops a stopping rule for gradient methods applied to non-convex problems with inexact gradients, ensuring solution quality and proximity to the initial point, supported by theoretical analysis and computational experiments.

Contribution

It introduces a novel early stopping rule for gradient methods that guarantees solution quality and stability in non-convex, noisy settings, including adaptive step size variants.

Findings

The stopping rule ensures acceptable solution quality.

The rule maintains a moderate distance from the initial point.

Computational experiments confirm effectiveness of the stopping rule.

Abstract

We study the gradient method under the assumption that an additively inexact gradient is available for, generally speaking, non-convex problems. The non-convexity of the objective function, as well as the use of an inexactness specified gradient at iterations, can lead to various problems. For example, the trajectory of the gradient method may be far enough away from the starting point. On the other hand, the unbounded removal of the trajectory of the gradient method in the presence of noise can lead to the removal of the trajectory of the method from the desired exact solution. The results of investigating the behavior of the trajectory of the gradient method are obtained under the assumption of the inexactness of the gradient and the condition of gradient dominance. It is well known that such a condition is valid for many important non-convex problems. Moreover, it leads to good complexity guarantees for the gradient method. A rule of early stopping of the gradient method is proposed. Firstly, it guarantees achieving an acceptable quality of the exit point of the method in terms of the function. Secondly, the stopping rule ensures a fairly moderate distance of this point from the chosen initial position. In addition to the gradient method with a constant step, its variant with adaptive step size is also investigated in detail, which makes it possible to apply the developed technique in the case of an unknown Lipschitz constant for the gradient. Some computational experiments have been carried out which demonstrate effectiveness of the proposed stopping rule for the investigated gradient methods.