Memory-Query Tradeoffs for Randomized Convex Optimization

TL;DR

This paper establishes fundamental memory-query tradeoffs for randomized convex optimization algorithms, showing that optimal performance requires either high memory or many queries, with cutting plane methods being Pareto-optimal.

Contribution

It proves that any randomized first-order method must trade off between memory and query complexity, demonstrating the optimality of cutting plane methods in this context.

Findings

Cutting plane methods are Pareto-optimal among first-order algorithms.

Quadratic memory is necessary for optimal query complexity.

Tradeoffs depend on the dimension and precision parameters.

Abstract

We show that any randomized first-order algorithm which minimizes a $d$-dimensional, $1$-Lipschitz convex function over the unit ball must either use $\Omega(d^{2-\delta})$ bits of memory or make $\Omega(d^{1+\delta/6-o(1)})$ queries, for any constant $\delta\in (0,1)$ and when the precision $\epsilon$ is quasipolynomially small in $d$. Our result implies that cutting plane methods, which use $\tilde{O}(d^2)$ bits of memory and $\tilde{O}(d)$ queries, are Pareto-optimal among randomized first-order algorithms, and quadratic memory is required to achieve optimal query complexity for convex optimization.