Convex optimization with $p$-norm oracles

TL;DR

This paper demonstrates that solving smoothed $ ext{l}_p$-norm problems can accelerate $ ext{l}_s$-regression solutions for $2 \,\leq\, p < s$, by establishing new iteration bounds and optimal rates for convex optimization with $ ext{l}_p^s$-proximal oracles.

Contribution

It introduces improved accelerated convex optimization rates using $ ext{l}_p^s$-proximal oracles and applies these to enhance $ ext{l}_s$-regression solving efficiency.

Findings

Achieves $ ilde{O}(n^{\frac{\nu}{1+\nu}})$ iteration complexity for $ ext{l}_s$-regression.

Provides optimal rates for algorithms using $ ext{l}_p^s$-proximal oracles.

Extends techniques to high-order and quasi-self-concordant optimization.

Abstract

In recent years, there have been significant advances in efficiently solving $\ell_s$-regression using linear system solvers and $\ell_2$-regression [Adil-Kyng-Peng-Sachdeva, J. ACM'24]. Would efficient smoothed $\ell_p$-norm solvers lead to even faster rates for solving $\ell_s$-regression when $2 \leq p < s$? In this paper, we give an affirmative answer to this question and show how to solve $\ell_s$-regression using $\tilde{O}(n^{\frac{\nu}{1+\nu}})$ iterations of solving smoothed $\ell_p$ regression problems, where $\nu := \frac{1}{p} - \frac{1}{s}$. To obtain this result, we provide improved accelerated rates for convex optimization problems when given access to an $\ell_p^s(\lambda)$-proximal oracle, which, for a point $c$, returns the solution of the regularized problem $\min_{x} f(x) + \lambda ||x-c||_p^s$. Additionally, we show that these rates for the $\ell_p^s(\lambda)$-proximal oracle are optimal for algorithms that query in the span of the outputs of the oracle, and we further apply our techniques to settings of high-order and quasi-self-concordant optimization.