Fast adaptive by constants of strong-convexity and Lipschitz for gradient first order methods

TL;DR

This paper introduces adaptive first-order methods for convex optimization that automatically estimate strong convexity and Lipschitz constants, removing the need for prior knowledge of these parameters, thus enhancing practical applicability.

Contribution

It proposes new adaptive algorithms, ACGM and ALGM, based on restarts and constant estimation, improving the usability of optimal gradient methods in real-world tasks.

Findings

Algorithms are theoretically optimal in complexity.

Experimental results confirm practical efficiency.

Methods adaptively estimate constants without prior knowledge.

Abstract

The work is devoted to the construction of efficient and applicable to real tasks first-order methods of convex optimization, that is, using only values of the target function and its derivatives. Construction uses OGM-G, fast gradient method which is optimal by complexity, but requires to know the Lipschitz constant for gradient and the strong convexity constant to determine the number of steps and step length. This requirement makes practical usage impossible. An adaptive on the constant for strong convexity algorithm ACGM is proposed, based on restarts of the OGM-G with update of the strong convexity constant estimate, and an adaptive on the Lipschitz constant for gradient ALGM, in which the use of OGM-G restarts is supplemented by the selection of the Lipschitz constant with verification of the convexity conditions used in the universal gradient descent method. This eliminates the disadvantages of the original method associated with the need to know these constants, which makes practical usage possible. Optimality of estimates for the complexity of the constructed algorithms is proved. To verify the results obtained, experiments on model functions and real tasks from machine learning are carried out.