OPTAMI: Global Superlinear Convergence of High-order Methods

TL;DR

This paper advances high-order convex optimization by establishing global superlinear convergence for certain methods, introducing a practical accelerated tensor method, and providing an open-source library to facilitate their application.

Contribution

It proves superlinear convergence of high-order methods for star-convex functions, introduces NATA, a practical accelerated tensor method, and releases OPTAMI, a comprehensive library for high-order optimization.

Findings

High-order methods exhibit global superlinear convergence for star-convex functions.

NATA significantly outperforms classical high-order acceleration methods.

The OPTAMI library enables practical application and research of high-order optimization methods.

Abstract

Second-order methods for convex optimization outperform first-order methods in terms of theoretical iteration convergence, achieving rates up to $O(k^{-5})$ for highly-smooth functions. However, their practical performance and applications are limited due to their multi-level structure and implementation complexity. In this paper, we present new results on high-order optimization methods, supported by their practical performance. First, we show that the basic high-order methods, such as the Cubic Regularized Newton Method, exhibit global superlinear convergence for $\mu$-strongly star-convex functions, a class that includes $\mu$-strongly convex functions and some non-convex functions. Theoretical convergence results are both inspired and supported by the practical performance of these methods. Secondly, we propose a practical version of the Nesterov Accelerated Tensor method, called NATA. It significantly outperforms the classical variant and other high-order acceleration techniques in practice. The convergence of NATA is also supported by theoretical results. Finally, we introduce an open-source computational library for high-order methods, called OPTAMI. This library includes various methods, acceleration techniques, and subproblem solvers, all implemented as PyTorch optimizers, thereby facilitating the practical application of high-order methods to a wide range of optimization problems. We hope this library will simplify research and practical comparison of methods beyond first-order.