History-Aware Adaptive High-Order Tensor Regularization

TL;DR

This paper introduces a history-aware adaptive regularization method for optimizing composite functions, achieving optimal complexity guarantees without prior knowledge of Lipschitz constants, and demonstrates its effectiveness through numerical experiments.

Contribution

The paper proposes a novel adaptive regularization technique that uses historical Lipschitz estimates, matching standard tensor method complexities without requiring known Lipschitz constants.

Findings

Method matches complexity guarantees of standard tensor methods.

Achieves $ ilde{O}( ext{accuracy}^{-1/p})$ iteration complexity for convex functions.

Numerical experiments confirm the effectiveness of the adaptive approach.

Abstract

In this paper, we develop a new adaptive regularization method for minimizing a composite function, which is the sum of a $p$th-order ($p \ge 1$) Lipschitz continuous function and a simple, convex, and possibly nonsmooth function. We use a history of local Lipschitz estimates to adaptively select the current regularization parameter, an approach we shall term the {\it history-aware adaptive regularization method}. We explore how the selection of an appropriate volume of historical information affects both the theoretical and practical performance. By using all the historical information, our method matches the complexity guarantees of the standard $p$th-order tensor methods that require a known Lipschitz constant, for both convex and nonconvex objectives. In the nonconvex case, the number of iterations required to find an $(\epsilon_g,\epsilon_H)$-approximate second-order stationary point is bounded by $\mathcal{O}(\max\{\epsilon_g^{-(p+1)/p}, \epsilon_H^{-(p+1)/(p-1)}\})$. For convex functions, we establish an $\mathcal{O}(\epsilon^{-1/p})$ iteration complexity for finding an $\epsilon$-approximate optimal point and further propose an accelerated variant attaining an iteration complexity of $\mathcal{O}(\epsilon^{-1/(p+1)})$. For practical consideration, we propose several variants of this method with only part of historical information. We introduce cyclic and sliding-window strategies for choosing historical Lipschitz estimates, which mitigate the limitation of overly conservative updates. As long as a rough upper bound of the Lipschitz constant is known, these two variants achieve the same iteration complexity guarantees in terms of the input accuracy as the method using full historical information. Finally, extensive numerical experiments are conducted to demonstrate the effectiveness of our adaptive approach.