AdaDelay: Delay Adaptive Distributed Stochastic Convex Optimization

TL;DR

AdaDelay introduces delay-sensitive step size adjustments in distributed stochastic convex optimization, enabling faster initial convergence and improved overall performance in real-world, heterogeneous network environments.

Contribution

The paper proposes a novel delay-adaptive method for distributed stochastic convex optimization that adjusts stepsizes based on actual delays, improving convergence speed.

Findings

Enhanced convergence speed in experiments with real datasets.

Effective handling of heterogeneous delays in distributed networks.

Maintains asymptotic complexity despite adaptive stepsizes.

Abstract

We study distributed stochastic convex optimization under the delayed gradient model where the server nodes perform parameter updates, while the worker nodes compute stochastic gradients. We discuss, analyze, and experiment with a setup motivated by the behavior of real-world distributed computation networks, where the machines are differently slow at different time. Therefore, we allow the parameter updates to be sensitive to the actual delays experienced, rather than to worst-case bounds on the maximum delay. This sensitivity leads to larger stepsizes, that can help gain rapid initial convergence without having to wait too long for slower machines, while maintaining the same asymptotic complexity. We obtain encouraging improvements to overall convergence for distributed experiments on real datasets with up to billions of examples and features.