Gradient-Type Methods for Optimization Problems with Polyak-{\L}ojasiewicz Condition: Early Stopping and Adaptivity to Inexactness Parameter

TL;DR

This paper introduces a fully adaptive gradient method for optimization problems satisfying the Polyak-Łojasiewicz condition, which adjusts to both the Lipschitz constant and gradient inexactness, improving convergence analysis and practical performance.

Contribution

It proposes a novel fully adaptive gradient algorithm that adapts to gradient Lipschitz constant and noise level, with detailed convergence analysis and empirical validation.

Findings

The algorithm converges under the Polyak-Łojasiewicz condition.

Numerical experiments demonstrate improved performance over non-adaptive methods.

The method effectively handles inexact gradient information.

Abstract

Due to its applications in many different places in machine learning and other connected engineering applications, the problem of minimization of a smooth function that satisfies the Polyak-{\L}ojasiewicz condition receives much attention from researchers. Recently, for this problem, the authors of recent work proposed an adaptive gradient-type method using an inexact gradient. The adaptivity took place only with respect to the Lipschitz constant of the gradient. In this paper, for problems with the Polyak-{\L}ojasiewicz condition, we propose a full adaptive algorithm, which means that the adaptivity takes place with respect to the Lipschitz constant of the gradient and the level of the noise in the gradient. We provide a detailed analysis of the convergence of the proposed algorithm and an estimation of the distance from the starting point to the output point of the algorithm. Numerical experiments and comparisons are presented to illustrate the advantages of the proposed algorithm in some examples.