Optimal Time Complexities of Parallel Stochastic Optimization Methods Under a Fixed Computation Model

TL;DR

This paper establishes the fundamental limits and optimal algorithms for parallel stochastic optimization under a fixed computation time model, extending classical sequential theory to parallel settings.

Contribution

It introduces a new protocol generalizing the oracle framework and derives minimax complexities for parallel methods with stochastic gradients, providing optimal algorithms.

Findings

Derived minimax complexity bounds for parallel stochastic optimization.

Developed algorithms that attain these bounds, proving their optimality.

Revealed implications for asynchronous optimization methods.

Abstract

Parallelization is a popular strategy for improving the performance of iterative algorithms. Optimization methods are no exception: design of efficient parallel optimization methods and tight analysis of their theoretical properties are important research endeavors. While the minimax complexities are well known for sequential optimization methods, the theory of parallel optimization methods is less explored. In this paper, we propose a new protocol that generalizes the classical oracle framework approach. Using this protocol, we establish minimax complexities for parallel optimization methods that have access to an unbiased stochastic gradient oracle with bounded variance. We consider a fixed computation model characterized by each worker requiring a fixed but worker-dependent time to calculate stochastic gradient. We prove lower bounds and develop optimal algorithms that attain them. Our results have surprising consequences for the literature of asynchronous optimization methods.