Lower Bounds on the Oracle Complexity of Nonsmooth Convex Optimization via Information Theory

TL;DR

This paper introduces an information-theoretic method to establish tight lower bounds on the oracle complexity of nonsmooth convex optimization, unifying previous techniques and covering various regimes and complexity measures.

Contribution

It presents a unified framework using information theory to derive tight lower bounds on oracle complexity for nonsmooth convex optimization, covering distributional, high-probability, and bounded-error complexities.

Findings

Established tight lower bounds for the box $[-1,1]^n$ and $L^p$-balls for $p \,\geq\, 1$

Unified analysis for worst-case, randomized, high-probability, and bounded-error complexities

Closed the gap between distributional and worst-case oracle complexities

Abstract

We present an information-theoretic approach to lower bound the oracle complexity of nonsmooth black box convex optimization, unifying previous lower bounding techniques by identifying a combinatorial problem, namely string guessing, as a single source of hardness. As a measure of complexity we use distributional oracle complexity, which subsumes randomized oracle complexity as well as worst-case oracle complexity. We obtain strong lower bounds on distributional oracle complexity for the box $[-1,1]^n$, as well as for the $L^p$-ball for $p \geq 1$ (for both low-scale and large-scale regimes), matching worst-case upper bounds, and hence we close the gap between distributional complexity, and in particular, randomized complexity, and worst-case complexity. Furthermore, the bounds remain essentially the same for high-probability and bounded-error oracle complexity, and even for combination of the two, i.e., bounded-error high-probability oracle complexity. This considerably extends the applicability of known bounds.