Nearly optimal first-order methods for convex optimization under gradient norm measure: An adaptive regularization approach

TL;DR

This paper introduces an adaptive regularization method for convex optimization that achieves nearly optimal iteration complexity without prior knowledge of the solution distance, and adapts to error bound conditions for faster convergence.

Contribution

It proposes a novel adaptive regularization approach for first-order convex optimization methods that does not require prior knowledge of the solution distance and adapts to error bound conditions.

Findings

Achieves nearly optimal iteration complexity without prior distance knowledge.

Adapts to H"olderian error bounds for faster convergence.

Applicable to a wide class of convex optimization problems.

Abstract

In the development of first-order methods for smooth (resp., composite) convex optimization problems, where smooth functions with Lipschitz continuous gradients are minimized, the gradient (resp., gradient mapping) norm becomes a fundamental optimality measure. Under this measure, a fixed iteration algorithm with the optimal iteration complexity is known, while determining this number of iteration to obtain a desired accuracy requires the prior knowledge of the distance from the initial point to the optimal solution set. In this paper, we report an adaptive regularization approach, which attains the nearly optimal iteration complexity without knowing the distance to the optimal solution set. To obtain further faster convergence adaptively, we secondly apply this approach to construct a first-order method that is adaptive to the H\"olderian error bound condition (or equivalently, the {\L}ojasiewicz gradient property), which covers moderately wide classes of applications. The proposed method attains nearly optimal iteration complexity with respect to the gradient mapping norm.