Efficient Convex Optimization Requires Superlinear Memory

TL;DR

This paper proves that memory-limited first-order convex optimization algorithms require significantly more queries than optimal methods, establishing a fundamental trade-off between memory and query complexity.

Contribution

It establishes a lower bound on the query complexity of memory-constrained convex optimization algorithms, resolving an open problem from COLT 2019.

Findings

Memory-constrained algorithms need at least rac{d^{1+(4/3)\u03b4}}{} queries.

Optimal cutting plane methods use rac{d^2}{} memory and rac{d}{} queries.

Memory limitations cause polynomial degradation in optimization performance.

Abstract

We show that any memory-constrained, first-order algorithm which minimizes $d$-dimensional, $1$-Lipschitz convex functions over the unit ball to $1/\mathrm{poly}(d)$ accuracy using at most $d^{1.25 - \delta}$ bits of memory must make at least $\tilde{\Omega}(d^{1 + (4/3)\delta})$ first-order queries (for any constant $\delta \in [0, 1/4]$). Consequently, the performance of such memory-constrained algorithms are a polynomial factor worse than the optimal $\tilde{O}(d)$ query bound for this problem obtained by cutting plane methods that use $\tilde{O}(d^2)$ memory. This resolves a COLT 2019 open problem of Woodworth and Srebro.