Faster Acceleration for Steepest Descent

TL;DR

This paper introduces a new accelerated first-order method for convex optimization under non-Euclidean smoothness, achieving significant iteration complexity improvements for lp smooth functions, overcoming longstanding barriers in non-Euclidean steepest descent.

Contribution

It proposes a novel primal-dual approach using differing norms and an implicit interpolation parameter, advancing acceleration techniques for lp smooth convex functions.

Findings

Achieves up to O(d^{1-rac{2}{p}}) iteration complexity improvement.

Circumvents previous barriers in accelerated non-Euclidean steepest descent.

Applicable to high-dimensional lp smooth convex optimization problems.

Abstract

Recent advances (Sherman, 2017; Sidford and Tian, 2018; Cohen et al., 2021) have overcome the fundamental barrier of dimension dependence in the iteration complexity of solving $\ell_\infty$ regression with first-order methods. Yet it remains unclear to what extent such acceleration can be achieved for general $\ell_p$ smooth functions. In this paper, we propose a new accelerated first-order method for convex optimization under non-Euclidean smoothness assumptions. In contrast to standard acceleration techniques, our approach uses primal-dual iterate sequences taken with respect to $\textit{differing}$ norms, which are then coupled using an $\textit{implicitly}$ determined interpolation parameter. For $\ell_p$ norm smooth problems in $d$ dimensions, our method provides an iteration complexity improvement of up to $O(d^{1-\frac{2}{p}})$ in terms of calls to a first-order oracle, thereby allowing us to circumvent long-standing barriers in accelerated non-Euclidean steepest descent.