Delayed Stochastic Algorithms for Distributed Weakly Convex Optimization

TL;DR

This paper introduces improved delayed stochastic algorithms for distributed weakly convex optimization, achieving tighter convergence rates that depend on expected delays and demonstrating robustness against arbitrary delays.

Contribution

It develops new delayed stochastic algorithms with convergence rates depending on expected delays and introduces a safeguarding step to eliminate delay effects, advancing distributed weakly convex optimization.

Findings

Delayed stochastic subgradient method has a tighter convergence rate based on expected delay.

The new delayed stochastic prox-linear method reduces delay impact to high-order terms.

Algorithms are robust against arbitrary delays and outperform existing methods in experiments.

Abstract

This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node. Recently, Xu et al. 2022 showed that an inertial stochastic subgradient method converges at a rate of $\mathcal{O}(\tau_{\text{max}}/\sqrt{K})$ which depends on the maximum information delay $\tau_{\text{max}}$. In this work, we show that the delayed stochastic subgradient method ($\texttt{DSGD}$) obtains a tighter convergence rate which depends on the expected delay $\bar{\tau}$. Furthermore, for an important class of composition weakly convex problems, we develop a new delayed stochastic prox-linear ($\texttt{DSPL}$) method in which the delays only affect the high-order term in the complexity rate and hence, are negligible after a certain number of $\texttt{DSPL}$ iterations. In addition, we demonstrate the robustness of our proposed algorithms against arbitrary delays. By incorporating a simple safeguarding step in both methods, we achieve convergence rates that depend solely on the number of workers, eliminating the effect of the delay. Our numerical experiments further confirm the empirical superiority of our proposed methods.