High-order Accumulative Regularization for Gradient Minimization in Convex Programming

TL;DR

This paper introduces a unified high-order accumulative regularization framework that accelerates gradient norm convergence in convex optimization, matching the speed of function-value residual reduction, and extends to parameter-free, inexact, uniformly convex settings.

Contribution

It develops a parameter-free high-order regularization method that achieves fast gradient norm convergence in convex and uniformly convex problems, unifying and generalizing prior approaches.

Findings

Matches the best convergence rates for function-value residuals.

Achieves linear, superlinear, and sublinear convergence in uniformly convex cases.

Requires no problem-specific parameter inputs.

Abstract

This paper develops a unified high-order accumulative regularization (AR) framework for convex and uniformly convex gradient norm minimization. Existing high-order methods often exhibit a gap: the function-value residual decreases fast, while the gradient norm converges much slower. To close this gap, we introduce AR that systematically transforms the fast function-value residual convergence rate into a fast (matching) gradient norm convergence rate. Specifically, for composite convex problems, to compute an approximate solution such that the norm of its (sub)gradient does not exceed $\varepsilon,$ the proposed AR methods match the best corresponding convergence rate for the function-value residual. We further extend the framework to uniformly convex settings, establishing linear, superlinear, and sublinear convergence of the gradient norm under different lower curvature conditions. Moreover, we design parameter-free algorithms that require no input of problem parameters, e.g., the Lipschitz constant of the $p$-th-order gradient, the initial optimality gap and the uniform convexity parameter, and allow an inexact solution for each high-order step. To the best of our knowledge, no parameter-free methods can attain such a fast gradient norm convergence rate which matches that of the function-value residual in the convex case, and no such parameter-free methods for uniformly convex problems exist. These results substantially generalize existing parameter-free and inexact high-order methods and recover first-order algorithms as special cases, providing a unified approach for fast gradient minimization across a broad range of smoothness and curvature regimes.