A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order

TL;DR

This paper offers a comprehensive analysis of the linear speedup potential in asynchronous parallel stochastic optimization algorithms, covering both zeroth- and first-order methods, and introduces a novel asynchronous zeroth-order method with practical applications.

Contribution

It provides a unified analysis framework for asynchronous stochastic algorithms, improves existing results, and proposes a new asynchronous zeroth-order method with demonstrated applications.

Findings

Recovers and enhances existing speedup analyses.

Provides new insights into asynchronous parallel behaviors.

Introduces a novel asynchronous zeroth-order method.

Abstract

Asynchronous parallel optimization received substantial successes and extensive attention recently. One of core theoretical questions is how much speedup (or benefit) the asynchronous parallelization can bring us. This paper provides a comprehensive and generic analysis to study the speedup property for a broad range of asynchronous parallel stochastic algorithms from the zeroth order to the first order methods. Our result recovers or improves existing analysis on special cases, provides more insights for understanding the asynchronous parallel behaviors, and suggests a novel asynchronous parallel zeroth order method for the first time. Our experiments provide novel applications including model blending problems using the proposed asynchronous parallel zeroth order method.