Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Server

TL;DR

This paper introduces an asynchronous optimization algorithm within a parameter server framework, providing convergence guarantees and practical validation through simulations and cloud implementations.

Contribution

It presents a novel asynchronous incremental aggregated gradient algorithm with explicit convergence analysis for regularized convex optimization.

Findings

Linear convergence rate established for strongly convex data loss

Explicit step-size expressions guaranteeing convergence

Validation through simulations and cloud-based implementations

Abstract

This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss is strongly convex, we establish linear convergence rate, give explicit expressions for step-size choices that guarantee convergence to the optimum, and bound the associated convergence factors. The expressions have an explicit dependence on the degree of asynchrony and recover classical results under synchronous operation. Simulations and implementations on commercial compute clouds validate our findings.