Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization

TL;DR

This paper introduces a unified framework for analyzing parallel stochastic optimization, deriving lower bounds based on dependency graphs, and identifying gaps between theoretical limits and existing algorithms.

Contribution

It presents a general oracle-based framework for parallel stochastic optimization and establishes lower bounds for various settings, revealing gaps and open questions.

Findings

Derived lower bounds for multiple parallel optimization scenarios

Identified gaps between lower bounds and existing algorithms

Highlighted cases where optimality of natural algorithms is unknown

Abstract

We suggest a general oracle-based framework that captures different parallel stochastic optimization settings described by a dependency graph, and derive generic lower bounds in terms of this graph. We then use the framework and derive lower bounds for several specific parallel optimization settings, including delayed updates and parallel processing with intermittent communication. We highlight gaps between lower and upper bounds on the oracle complexity, and cases where the "natural" algorithms are not known to be optimal.