Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delays

TL;DR

This paper introduces a robust stochastic convex optimization method that adaptively handles asynchronous gradient delays without prior knowledge, suitable for dynamic shared-resource environments.

Contribution

The proposed method is the first to implicitly adapt to changing gradient delays in asynchronous optimization without requiring prior delay or smoothness information.

Findings

Provides non-asymptotic convergence guarantees.

Handles dynamic and unknown gradient delays effectively.

Suitable for cloud and data center environments.

Abstract

We consider stochastic convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory. We propose a robust training method for the constrained setting and derive non asymptotic convergence guarantees that do not depend on prior knowledge of update delays, objective smoothness, and gradient variance. Conversely, existing methods for this setting crucially rely on this prior knowledge, which render them unsuitable for essentially all shared-resources computational environments, such as clouds and data centers. Concretely, existing approaches are unable to accommodate changes in the delays which result from dynamic allocation of the machines, while our method implicitly adapts to such changes.