Lower Complexity Bounds for Minimizing Regularized Functions

TL;DR

This paper establishes fundamental lower bounds on the convergence rates of first-order methods for regularized convex optimization, demonstrating optimality of certain accelerated algorithms and covering various regularization and norm cases.

Contribution

The paper derives new lower complexity bounds for first-order methods on regularized convex functions with different norms, establishing optimal rates and covering non-Euclidean cases.

Findings

Optimal rate for Euclidean norms is O(k^{-p(1+3ν)/(2(p-1-ν))})

Fast Gradient Method achieves the optimal rate of O(k^{-6}) in cubic regularization cases

Lower bounds are also established for non-Euclidean norms.

Abstract

In this paper, we establish lower bounds for the oracle complexity of the first-order methods minimizing regularized convex functions. We consider the composite representation of the objective. The smooth part has H\"older continuous gradient of degree $\nu \in [0, 1]$ and is accessible by a black-box local oracle. The composite part is a power of a norm. We prove that the best possible rate for the first-order methods in the large-scale setting for Euclidean norms is of the order $O(k^{- p(1 + 3\nu) / (2(p - 1 - \nu))})$ for the functional residual, where $k$ is the iteration counter and $p$ is the power of regularization. Our formulation covers several cases, including computation of the Cubically regularized Newton step by the first-order gradient methods, in which case the rate becomes $O(k^{-6})$. It can be achieved by the Fast Gradient Method. Thus, our result proves the latter rate to be optimal. We also discover lower complexity bounds for non-Euclidean norms.