The global rate of convergence for optimal tensor methods in smooth convex optimization

TL;DR

This paper introduces a new tensor optimization method that achieves the optimal convergence rate for smooth convex functions with high-order derivatives, improves theoretical bounds, and demonstrates practical efficiency over existing methods.

Contribution

The paper proposes a novel tensor method that attains the optimal convergence rate for convex optimization with high-order derivatives and introduces a new condition number for analysis.

Findings

The new method closes the gap between lower and upper iteration complexity bounds.

Accelerated convergence is achieved for uniformly convex functions.

Numerical experiments show the third-order method outperforms the second-order method in practice.

Abstract

We consider convex optimization problems with the objective function having Lipshitz-continuous $p$-th order derivative, where $p\geq 1$. We propose a new tensor method, which closes the gap between the lower $O\left(\varepsilon^{-\frac{2}{3p+1}} \right)$ and upper $O\left(\varepsilon^{-\frac{1}{p+1}} \right)$ iteration complexity bounds for this class of optimization problems. We also consider uniformly convex functions, and show how the proposed method can be accelerated under this additional assumption. Moreover, we introduce a $p$-th order condition number which naturally arises in the complexity analysis of tensor methods under this assumption. Finally, we make a numerical study of the proposed optimal method and show that in practice it is faster than the best known accelerated tensor method. We also compare the performance of tensor methods for $p=2$ and $p=3$ and show that the 3rd-order method is superior to the 2nd-order method in practice.