The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory

TL;DR

This paper analyzes the convergence of asynchronous stochastic gradient descent in shared-memory systems, providing new bounds and revealing a trade-off between delay and convergence rate, thus enhancing understanding of this fundamental algorithm in concurrent environments.

Contribution

It offers the first convergence bounds for lock-free asynchronous SGD in shared memory, improving existing bounds and identifying key trade-offs affecting efficiency.

Findings

SGD can converge faster in asynchronous shared memory than previously thought.

A fundamental trade-off exists between system delay and convergence speed.

New bounds extend the parameter range for effective asynchronous SGD.

Abstract

Stochastic Gradient Descent (SGD) is a fundamental algorithm in machine learning, representing the optimization backbone for training several classic models, from regression to neural networks. Given the recent practical focus on distributed machine learning, significant work has been dedicated to the convergence properties of this algorithm under the inconsistent and noisy updates arising from execution in a distributed environment. However, surprisingly, the convergence properties of this classic algorithm in the standard shared-memory model are still not well-understood. In this work, we address this gap, and provide new convergence bounds for lock-free concurrent stochastic gradient descent, executing in the classic asynchronous shared memory model, against a strong adaptive adversary. Our results give improved upper and lower bounds on the "price of asynchrony" when executing the fundamental SGD algorithm in a concurrent setting. They show that this classic optimization tool can converge faster and with a wider range of parameters than previously known under asynchronous iterations. At the same time, we exhibit a fundamental trade-off between the maximum delay in the system and the rate at which SGD can converge, which governs the set of parameters under which this algorithm can still work efficiently.

Figures

Equations

$$x_{t+1}=x_t-{\alpha }_t\tilde{g}(x_t),$$

$$f(x)=\frac{1}{m}\sum _{i=1}^m{L}_i(x).$$

$$E[\tilde{g}(x)]=\frac{1}{m}\sum _{i=1}^m\nabla L_i(x)=\nabla f(x).$$

$$x_{t+1}=x_t-{\alpha }_t\tilde{g}(x_t),$$

$${\left(x-y\right)}^T\left(\nabla f\left(x\right)-\nabla f\left(y\right)\right)\ge c\Vert x-y{\Vert }^2$$

$$\mathbb{E}\left(\Vert \tilde{g}\left(x\right)-\tilde{g}\left(y\right)\Vert \right)\le L\Vert x-y\Vert$$

$$\mathbb{E}\left(\Vert \tilde{g}\left(x\right){\Vert }^2\right)\le M^2.$$

$$S=\{x|\Vert x-x^{\ast }{\Vert }^2\le ϵ\}$$

$$\mathbb{P}\left(F_T\right)\le \frac{M^2}{c^2ϵ\vartheta T}\log \left(e\Vert x_0-x^{\ast }{\Vert }^2{ϵ}^{-1}\right).$$

$$f(x)=\frac{1}{2}{x}^2,$$

$$\tilde{g}(x)=x-\tilde{u},$$

$$x_{t+1}=x_t-\alpha \tilde{g}(x_t)=(1-\alpha )x_t+\alpha {\tilde{u}}_t.$$

$$x_{\tau }=(1-\alpha {)}^{\tau }{x}_0+\alpha \sum _{k=1}^{\tau }(1-\alpha {)}^{\tau -k}{\tilde{u}}_{k-1}.$$

$$x_{\tau +1}=\left((1-\alpha {)}^{\tau }-\alpha \right)x_0+\alpha \sum _{k=1}^{\tau }(1-\alpha {)}^{\tau -k}{\tilde{u}}_{k-1}+\alpha {\tilde{u}}_{\tau }.$$

$${\overline{\sigma }}^2$$

$$={\alpha }^2{\sigma }^2\left(1+\sum _{k=0}^{\tau -1}{\left((1-\alpha {)}^2\right)}^k\right)$$

$$={\alpha }^2{\sigma }^2\left(1+\frac{1-(1-\alpha {)}^{2\tau }}{1-(1-\alpha {)}^2}\right).$$

$$\Vert x_{\tau +1}\Vert \ge \frac{\alpha }{2}\Vert x_0\Vert ,\text{ versus }\Vert x_{\tau +1}\Vert =(1-\alpha {)}^{\tau }\Vert x_0\Vert$$

$$\frac{\log \left((1-\alpha {)}^{\tau }\right)}{\log \left(\frac{\alpha }{2}\right)}=\tau \frac{\log (1-\alpha )}{\log (\alpha )-\log (2)}=\Omega (\tau ),$$

$$\mathbb{E}\left(W_{t+1}\left(x_t-\alpha \tilde{g}\left(x_t\right),x_t,...,x_0\right)\right)\le W_t\left(x_t,x_{t-1},...,x_0\right),$$

$$W_T\left(x_T,x_{T-1},...,x_0\right)\ge T.$$

$$\mathbb{P}\left(F_T\right)\le \frac{M^2+2LM\tau \sqrt{ϵ}}{c^2ϵ\vartheta T}\log \left(e\Vert x_0-x^{\ast }{\Vert }^2{ϵ}^{-1}\right).$$

$$\Vert x_t-v_t{\Vert }_1$$

$$\le \sum _{k=1}^t\Vert x_{t-k+1}-x_{t-k}{\Vert }_1\text{mathbbm}{1}_{\{{\tau }_t\ge k\}}$$

$$\Vert x_t-v_t\Vert \le \sqrt{d}\sum _{k=1}^t\Vert x_{t-k+1}-x_{t-k}\Vert \text{mathbbm}{1}_{\{{\tau }_t\ge k\}}$$

$$\sum _{m=1}^{\infty }\text{mathbbm}{1}_{\{{\tau }_{t+m}\ge m\}}=\sum _{m=1}^{{\tau }_{\text{max}}}\text{mathbbm}{1}_{\{{\tau }_{t+m}\ge m\}}\le 2\sqrt{{\tau }_{\text{max}}n}.$$

$$\sum _{m=1}^{{\tau }_{\text{max}}}\text{mathbbm}{1}_{\{{\tau }_{t+m}\ge m\}}$$

$$\le Kn+\left(\frac{{\tau }_{\text{max}}-Kn}{Kn}+1\right)n$$

$$=\frac{{\tau }_{\text{max}}}{K}+Kn.$$

$$\Vert W_t\left(u,x_{t-1},...,x_0\right)-W_t\left(v,x_{t-1},...,x_0\right)\Vert \le H\Vert u-v\Vert$$

$$\mathbb{P}\left(F_T\right)\le \frac{\mathbb{E}\left(W_0\left(x_0\right)\right)}{\left(1-{\alpha }^{2}HLMC\sqrt{d}\right)T}.$$

$$\displaystyle\textnormal{plog}(x)=\left\{\begin{array}[]{lr}\log(ex)&:x\geq 1\\ x&:x\leq 1\end{array}\right.$$

$$W_t(x_t,\dots ,x_0)=\frac{ϵ}{2\alpha cϵ-{\alpha }^2{M}^2}\text{plog}\left(\Vert x_t-x^{\ast }{\Vert }^2{ϵ}^{-1}\right)+t.$$

Full text

In this work, we address this gap, and provide new convergence bounds for lock-free concurrent stochastic gradient descent, executing in the classic asynchronous shared memory model, against a strong adaptive adversary. Our results give improved upper and lower bounds on the “price of asynchrony” when executing the fundamental SGD algorithm in a concurrent setting. They show that this classic optimization tool can converge faster and with a wider range of parameters than previously known under asynchronous iterations. At the same time, we exhibit a fundamental trade-off between the maximum delay in the system and the rate at which SGD can converge, which governs the set of parameters under which this algorithm can still work efficiently.

1 Introduction

Background.

The tremendous recent progress in machine learning can be explained in part through the availability of extremely vast amounts of data, but also through improved computational support for executing machine learning tasks at scale. Perhaps surprisingly, some of the main algorithms driving this progress are have been known in some form or another for a very long time. One such tool is the classic stochastic gradient descent (SGD) optimization algorithm [24], introduced by Robbins and Munro in the 1950s, variants of which are currently the tool of choice for optimization in a variety of settings, such as image classification and speech recognition via neural networks, but also in fundamental data processing tools such as regression.

In a nutshell, SGD works as follows. Given a $d$-dimensional function $f:\operatorname{\mathbb{R}}^{d}\rightarrow\operatorname{\mathbb{R}}$ which we want to minimize, we have access to stochastic gradients $\widetilde{g}(x)$ of this function at the point $x$, that is, random vectors with the property that $\operatorname{\mathbb{E}}[\widetilde{g}(x)]=\nabla f(x)$. SGD then consists in the iteration

[TABLE]

where $x_{t}$ is a $d$-dimensional vector which we will henceforth call the model, encoding our current beliefs about the data, and $\alpha_{t}$ is the step size, which controls how “aggressive” updates should be.

A classic instance of this method, which we consider in this paper, is the following. We are given a large set of $m$ data points $X_{1},\ldots,X_{m}$, where to each point $X_{i}$, with $i\in\{1,2,\ldots,m\}$, we associate a loss function $L_{i}(x)$, measuring the loss of any model $x$ at the data point $X_{i}$ (mapping the difference between the prediction of our model on $X_{i}$ with the true label to a real value), and we wish to identify a model $x^{\star}$ which minimizes the average loss over the dataset, that is, minimize the function

[TABLE]

This setting can be mapped to fundamental optimization tasks, such as linear regression over a given set of points, or training neural networks through backpropagation [25]. In both settings, we are given a large dataset, from which we pick a sample point uniformly at random in every iteration $t$. In every such iteration, a process reads the current version of the model $x_{t}$, computes the stochastic gradient $\widetilde{g}(x_{t})$ of this model with respect to the current sample by examining the derivative of the loss function at the point, given its label, and updates the model’s components correspondingly. Thus, the stochastic gradients $\widetilde{g}(x_{t})$ correspond to the gradient of the model $x_{t}$ taken at a uniformly random data point $X_{i}$. Indeed, since the data point at which we compute the gradient is chosen uniformly at random, we have

[TABLE]

Parallel SGD.

The ability to parallelize this process at extremely large scale, up to thousands of nodes, e.g. [11], has enabled researchers to approach new problems, and to reach super-human accuracy for several classic problems, e.g. [15, 29, 14, 28, 8]. The standard way to distribute SGD to multiple compute nodes is data parallelism: given $n$ nodes, which we will abstract as parallel processes, we split the dataset into $n$ partitions. Nodes process samples from their partitions in parallel, and synchronize by using a shared model $x$. In this paper, we will focus on the case where the nodes are threads, communicating through asynchronous shared memory.

In theory, data parallelism would allow the system to perform $n$ times more iterations (and model updates) per unit of time. The catch is that threads will need to synchronize on the shared model, reducing scalability. In fact, early studies [16] proposed using coarse-grained locking to keep the process consistent to a sequential execution. As expected, coarse-grained locking leads to significant loss of performance.

In this context, a breakthrough by Niu et al. [23] proved the unintuitive result that SGD should be able to converge even without coarse-grained synchronization to maintain model consistency, under some technical conditions, including high sparsity of the gradient updates, and under the assumption that individual updates are applied via fetch-and-add synchronization operations. (This latter assumption appears necessary in general, since otherwise a delayed thread could completely obliterate all progress made up to some point, by overwriting the entire model, which resides in shared memory.) This work has sparked significant research on the convergence properties of asynchronous SGD, e.g. [26, 10, 19, 18, 12, 34], improving theoretical bounds and considering more general settings.

In a nutshell, these results show, under various delay models and analytic assumptions, that SGD can still converge if iterations are asynchronous, that is, they cannot be “serialized” in any meaningful way. At the same time, all known convergence upper bounds, e.g. [10, 18], have a linear dependence in $\tau_{\max}$, the maximum delay between the time where a gradient update is generated by any thread, and the point where the gradient is applied to the model.111In shared-memory parlance, $\tau_{\max}$ is upper bounded by the interval contention, where we define SGD iterations as individual operations. Intuitively, $\tau_{\max}$ is upper bounded by the number of iterations which can take steps between the start and end points of any fixed iteration. This means that asynchronous SGD will take $\tau_{\max}$ times more iterations to converge, compared to the synchronous variant. Since $\tau_{\max}$ is technically only upper bounded by the length of the execution, this upper bound appears extremely harsh. It is therefore natural to ask if this dependency is inherent, or whether it can be improved, yielding superior convergence rates for this classical algorithm.

Contribution.

In this paper, we address this problem. Our approach is to express data-parallel SGD in the classic asynchronous shared memory model, against a strong (adaptive) adversarial scheduler, which designs schedules to delay the algorithm from converging, with full knowledge of the algorithm and random coin flips. Under this formulation, our main results are as follows:

• •

We show that, under standard analytic assumptions, for convex objective functions $f$, there exists a simple variant of the classic SGD algorithm which still converges under this strong adversarial model.

• •

We prove that, under reasonable parameter settings, the convergence slowdown of SGD iterations due to asynchrony can be upper bounded by $\sqrt{\tau_{\max}n}$, where $\tau_{\max}$ is the maximum interval contention over all operations and $n$ is the number of threads. This result shows for the first time that the runtime dependence need not be linear in $\tau_{\max}$, even against a strong adversary, and is our main technical contribution.

• •

We prove that, in general, the adversary can cause a convergence slowdown that is linear in $\tau_{\max}$, if the algorithm does not decrease the step size $\alpha_{t}$ to offset the influence of stale gradient updates. This shows for the first time that the adversary can consistently and significantly slow down convergence by inducing delays, and that decreasing the step size (which is done by our algorithm) is in fact necessary for good convergence under asynchrony.

In sum, our results give new and improved upper and lower bounds on the “price of asynchrony” when executing the fundamental SGD algorithm in a concurrent setting. They show that this classic optimization tool can converge faster and with a wider range of parameters than previously known, under asynchronous iterations. At the same time, we exhibit a simple yet fundamental trade-off between the maximum delay in the system and the convergence rate of SGD, which governs the set of parameters under which SGD can still work efficiently.

Techniques and Related Work.

From the technical perspective, our results build upon martingale-based approaches for bounding the convergence of SGD, e.g. [9, 10]. These techniques complement the classic “regret” bounds for characterizing the convergence of SGD, e.g. [5]. We exploit martingale-based techniques in the asynchronous setting. To our knowledge, the only other work to employ such techniques for convex SGD is [10], whose results we significantly extend. Specifically, with respect to this reference, the main departures are that 1) we consider a more challenging adaptive adversarial model, as opposed to a stochastic scheduling model; 2) eliminate the requirement that gradients contain a single non-zero entry, thereby significantly expanding the applicability of the framework; 3) reduce the linear convergence dependency in $\tau_{\max}$ to one of the form $\sqrt{\tau_{\max}n}$; 4) prove lower bounds on the slowdown due to asynchrony.

There is an extremely vast literature studying the convergence properties of asynchronous optimization methods [23, 19, 18, 10, 12, 17, 22], as well as efficient parallel implementations, e.g. [27, 34], starting with seminal work by Bertsekas and Tsitsiklis [3]. A complete survey is beyond the scope of this paper, and we therefore focus on work that is directly related to ours. Reference [23] showed for the first time that, under strong analytical assumptions on sparsity and on the target loss function, asynchronous SGD can still converge, and that, moreover, the convergence rate can be similar to that of the baseline under further assumptions on the parameters. Agarwal and Duchi [1] showed that, under strong ordering assumptions, delayed gradient computation does not affect the convergence of SGD. Lian et al. [18] provided ergodic convergence rates for asynchronous SGD for non-convex objectives. Duchi et al. [12] considered a model similar to ours, and showed that the impact of any asynchrony on the rate at which the algorithm converges is negligible, under strong technical assumptions on the convex function $f$ to be optimized, on the structure of its optimum, and on the sampling noise. Concurrent work [17, 22] provides significantly more general analyses of iterative processes under asynchrony, covering several important optimization algorithms. With the exception of [12], which makes strong technical assumptions, all previous results for asynchronous SGD had a linear dependence in the maximum delay $\tau_{\max}$. We improve this dependency in this work.

There exists significant work on mitigating the effects of asynchrony in applied settings, e.g. [35, 36, 20]. A subset of these works are designed for a distributed shared memory setting, where it may be possible to examine the “staleness” of an update immediately before applying it, and adjust hyperparameters accordingly, and validated empirically. By contrast, we consider an adversarial setting, where the scheduler actively attempts to thwart the algorithm’s progress. Our lower bound applies to these works as well.

There has recently been significant work connecting machine learning and optimization with distributed computing. References [4, 33, 7] consider distributed SGD in a Byzantine adversarial setting, but in a message-passing system. In a series of papers [30, 32], Su and Vaidya have considered the problem of adding fault-tolerance to the problem of multi-agent optimization, as well as non-Bayesian optimization under asynchrony and crash failures [31].

2 Model

Asynchronous Shared Memory.

We consider a standard asynchronous shared-memory model [2], in which $n$ threads (or processes) $P_{1},\ldots,P_{n}$, communicate through atomic memory locations called registers, on which they perform atomic operations such as $\mathsf{read}$, $\mathsf{write}$, $\mathsf{compare\&swap}$ and $\mathsf{fetch\&add}$. In particular, the algorithm we consider employs atomic $\mathsf{read}$ and $\mathsf{fetch\&add}$ operations, which are now standard in mass-produced multiprocessors. The $\mathsf{fetch\&add}$ operation takes one argument, and returns the value of the register before the increment was performed, incrementing its value by the corresponding operand.

As is usual, we will assume a sequentially consistent memory model, in which once a thread returns from its invocation of a primitive (for example, a $\mathsf{fetch\&add}$), the value written by the thread is immediately applied to shared memory, and henceforth visible by other processors.

The Adversarial Scheduler.

Threads follow an algorithm composed of shared-memory steps and local computation, including random coin flips. The order of process steps is controlled by an adversarial entity we call the scheduler. Time is measured in terms of the number of shared-memory steps scheduled by the adversary. The adversary may choose to crash a set of at most $n-1$ processes by not scheduling them for the rest of the execution. A process that is not crashed at a certain step is correct, and if it never crashes then it takes an infinite number of steps in the execution. In the following, we assume a standard strong adversarial scheduler, which can see the results of the threads’ local coins when deciding the scheduling.

Contention Bound.

In the following, fix a (concurrent) SGD iteration $\theta$, and let $\rho(\theta)$ be the interval contention of the iteration $\theta$, defined as the number of SGD iterations which can execute concurrently with $\theta$. Let $\tau_{\max}$ be the upper bound over all these interval contention values, i.e. $\tau_{\max}=\max_{\theta}\rho(\theta)$. Let $\tau_{avg}$ be an upper bound on the average interval contention over all iterations during the (finite) execution of a program, i.e. $\tau_{avg}=1/T\sum_{1\leq\theta\leq T}\rho(\theta)$, where $T$ is the total number of iterations of the algorithm. It is known that $\tau_{avg}\leq 2n$, where $n$ is the number of threads [13].

3 Background on Stochastic Gradient Descent

Stochastic gradient descent (SGD) is a standard method in optimization. Let $f:\operatorname{\mathbb{R}}^{d}\to\operatorname{\mathbb{R}}$ be a function which we want to minimize. We have access to stochastic gradients $\widetilde{g}$ of $f$, such that $\operatorname{\mathbb{E}}[\widetilde{g}(x)]=\nabla f(x)$. SGD will start at a randomly chosen point $x_{0}$, and converge towards the minimum by iterating the procedure

[TABLE]

where $x_{t}$ is the current candidate, and $\alpha_{t}$ is a variable step-size parameter. Notably, this arises if we are given i.i.d. data points $X_{1},\ldots,X_{m}$ generated from an unknown distribution $D$, and a loss function $\ell(X,\theta)$, which measures the loss of the model $\theta$ at data point $X$. We wish to find a model $\theta^{*}$ which minimizes $f(\theta)=\operatorname{\mathbb{E}}_{X\sim D}[\ell(X,\theta)]$, the expected loss to the data.

Unless otherwise noted, we consider SGD for convex optimization and with a constant learning rate, that is $\alpha_{t}=\alpha$ for all $t$. We also make the following standard assumptions:

• •

The function $f$ is strongly convex with parameter $c$, i.e. for all $x,y\in\mathbb{R}^{d}$:

[TABLE]

• •

The stochastic gradient $\widetilde{g}\left(x\right)$ is $L$-Lipschitz continuous in expectation for some $L>0$, i.e. for all $x,y\in\mathbb{R}^{d}$:

[TABLE]

• •

The second moment of the stochastic gradients is bounded by some $M^{2}>0$, i.e. for all $x\in\mathbb{R}^{d}$:

[TABLE]

Classic approaches for analyzing the convergence of SGD attempt to bound the distance between the expected value of $f$ at the average of the currently generated iterates and the optimal value of the function (e.g. Theorem 6.3 in [5]), showing that this distance decreases linearly with the number of iterations.

Here we consider a different approach that aims at estimating the probability that the algorithm has failed to converge to a success region around the optimal parameter value after $T$ steps. To this end, we employ a martingale-based analysis of the algorithm, an approach that has recently become a popular tool for analyzing asynchronous optimization algorithms [6, 10]. Let $x^{*}$ be the minimizer of the function $f$. Given an $\epsilon>0$, we denote by

[TABLE]

the success region around this minimizer, to which we want to converge. Our analysis aims to bound the probability of the event $F_{T}$ that $x_{i}\not\in S$ for all $i\leq T$, i.e. the event that the algorithm has failed to hit the success region by time $T$.

An existing result of this type about the convergence of parallel SGD was derived in [10]. Under the non-standard additional assumption that each gradient update on the parameter value only effects a single entry of $x_{t}$ (i.e., that the stochastic gradients contain a single non-zero entry),222Our analysis eliminates this assumption, a result which may be of independent interest. one can obtain the following:

Theorem 3.1** ([10]).**

Consider the SGD algorithm under the assumptions above, run for $T$ steps, with a success region $S=\{x\text{ }|\text{ }\|x-x^{*}\|^{2}\leq\epsilon\}$ and with learning rate $\alpha=\frac{c\epsilon\vartheta}{M^{2}}$ for some constant $\vartheta\in\left(0,1\right)$. Then the probability of the event $F_{T}$ that $x_{i}\not\in S$ for all $i\leq T$ is:

[TABLE]

Note that this bound also decreases linearly with the number of iterations.

4 Lock-Free SGD in Shared-Memory

A standard way of parallelizing the SGD algorithm is to have multiple parallel threads execute the procedure in Equation 1. We assume a lock-free setting, in which threads share the set of parameters (model) $X[d]$, which they can read and update entry-wise (through $\mathsf{read}$ and $\mathsf{fetch\&add}$ operations) concurrently, without additional synchronization. Each thread executes the steps in Algorithm 1:

We emphasize that this modeling of the algorithm is standard: virtually all papers which analyze asynchronous SGD consider this formulation, e.g. [23, 10, 12]. Updates are assumed to occur via fetch&add, to avoid complete resets of the state by a delayed thread.

5 A Slowdown Lower Bound via Adversarial Delays

Setup.

We now provide a simple argument which yields a lower bound on the achievable speedup if the adversary can delay gradients by a large $\tau_{\max}$. This argument might also serve as a brisk hands-on introduction to SGD. We consider a standard setting, where two threads each have access to local gradient samples, and share the model. Assume we are trying to minimize the (convex) objective function

[TABLE]

and have access to its noisy gradients

[TABLE]

where $\tilde{u}$ is normally distributed noise with mean [math] and variance $\sigma^{2}$. Observe the minimum is at [math], and $\operatorname{\mathbb{E}}[\widetilde{g}(x)]=\nabla f(x)$. Now if $u_{t}$ is the stochastic gradient at time $t$, the SGD update is

[TABLE]

Adversarial Strategy.

Suppose that the adversary is executing the following strategy. First, both threads generate a gradient with respect to $x_{0}$. The first thread then executes for $\tau$ consecutive iterations, and then the second writes the state with a gradient from the initial value. Let us now analyze the rate at which this algorithm can converge.

Analysis.

After the first thread runs, the output will be

[TABLE]

After the second thread merges in its stale gradient, which has the form $x_{0}-\tilde{u}_{\tau}$, we have:

[TABLE]

The second term of this last expression will be a zero-mean Gaussian with a variance of

[TABLE]

For a fixed $\alpha$, if we choose $\tau$ large enough that $2(1-\alpha)^{\tau}\leq\alpha$, and suppose for simplicity that $\sigma=0$, then we can get

[TABLE]

in the case with no adversary. To compare these rates, we take the logarithm, and obtain a slowdown factor of

[TABLE]

which implies an $\Omega(\tau)$ factor slowdown is possible from a delay of $\tau$. This shows that with a maximum delay $\tau_{\max}=\tau$, the adversary can achieve an asymptotic slowdown that is linear in $\tau$. We conclude as follows.

Theorem 5.1**.**

Given an instance of the lock-free SGD Algorithm in 1 with fixed learning rate $\alpha$, there exists an adversarial strategy with maximum delay $\tau_{\max}=O(\log(\alpha)/\log(1-\alpha))$ such that the algorithm converges $\tau_{\max}$ times slower than the sequential variant.

6 Convergence Upper Bounds in Asynchronous Shared Memory

6.1 Preliminaries

Notation and Iteration Ordering.

We now introduce some notation, and a few basic claims about the above concurrent process. First, we define an order on the above iterations, performed possibly by distinct threads, by the time at which the iteration performs its first fetch&add operation, on the first model component $X[1]$. (Here we are using the sequential consistency property of the memory model.) This ordering induces a useful total order between iterations: for any integer $t\geq 1$, iteration $t$ is the $t$th iteration to complete its fetch&add on $X[1]$. We now note that all of these iterations up to iteration $t$ must have completed their computation, but may not have completed writing their updates to $X$ by the time when iteration $t+1$ starts. At most $n$ of these iterations may be incomplete at any given time. We formalize this as follows.

Lemma 6.1**.**

Let iteration $t$ be the $t$th iteration to update $X[1]$. This is a total order on the iterations. We say that an iteration is incomplete at a given point in the execution if it has performed its first update (on $X[1]$), but has not completed its last update (on $X[d]$). For any $t\geq 1$, at most $n$ iterations with indices $\leq t$ can be incomplete.

By assumption the maximal interval contention is bounded by $\tau_{\max}$. However, since at most $n$ threads can run at the same time, it is intuitive that the average contention should be $\mathcal{O}\left(n\right)$. We formalize this via the following:

Lemma 6.2**.**

Fix a parameter $K$, and an arbitrary time interval $I$ during which exactly $Kn$ consecutive SGD iterations start. We call an SGD iteration $\theta$ bad if more than $Kn$ iterations start between its start time and end time. Otherwise, an SGD iteration is good. Then, the number of bad iterations which complete during $I$ is less than $n$.

Proof.

Assume for contradiction that the number of bad iterations is $n$ or larger. Then, there must exist a thread $p$ which completes two of these bad iterations during this interval. Denote the second iteration by $\theta$. Since the two iterations by $p$ cannot be concurrent with each other, the maximum number of iterations which can be concurrent with $\theta$ (except itself) is at most $Kn-2$. Hence, $\theta$ cannot be bad, a contradiction. ∎

Inconsistent Views $v_{t}$ and Accumulators $x_{t}$.

Let $\widetilde{g}_{t}$ be the gradient vector which the corresponding thread executing iteration $t$ wishes to add to $X$, and $v_{t}$ be this thread’s view of the model $X$ when generating this gradient, composed out of the values the thread read for each entry. Importantly, we define the auxiliary (global) vector $x_{t}=\sum_{k=1}^{t}\widetilde{g}_{k}$, containing all of the updates which various started iterations wish to apply to the model $X$ up to $t$. We note that the view $v_{t}$ which iteration $t$ uses to generate its gradient update can consist of a possibly inconsistent set of updates across the various components of $X$. However, at all times 1) all of these updates must be contained in $x_{t}$, in the sense that they feature in $\widetilde{g}_{k}$ for some $k\leq t$; and 2) if we denote by $t^{\prime}$ the highest iteration whose update is read by $v_{t}$, we must have that there can be at most $\ell(\theta)$ updates with indices $<t^{\prime}$ which $v_{t}$ sees as incomplete during its scan. This is illustrated in Figure 1.

SGD Convergence.

We now analyze the convergence of SGD under this lock-free model. Following [10], we employ a martingale approach for proving convergence rates of SGD.

Given a probability space $\left(\Omega,\mathcal{F},\mathbb{R}\right)$, a real-valued stochastic process $W:\mathbb{N}\times\Omega\rightarrow\mathbb{R}$ is called a supermartingale if $\mathbb{E}\left(W_{t+1}|W_{t},W_{t-1},...,W_{0}\right)\leq W_{t}$ for all $t\geq 0$. Here we consider supermartingales of the form $W_{t}\left(x_{t},...,x_{0}\right)$, defined based on the state of the optimization algorithm. Intuitively, $W_{t}$ represents our unhappiness about the state of the algorithm at time $t$. More precisely, we have the following:

Definition 6.1** ([10]).**

Given a stochastic optimization algorithm, a non-negative process $W_{t}:\mathbb{R}^{d\times t}\rightarrow\mathbb{R}$ is a rate supermartingale with horizon $B$, if two conditions hold. Firstly, it is a supermartingale, i.e. for all $x_{t},...,x_{0}$ and for all $t\leq B$:

[TABLE]

where expectation is taken with respect to the randomness at time $t$ and conditional on the past. Secondly, for any time $T\leq B$ and any sequence $x_{T},...,x_{0}$, if the algorithm has not succeeded by time $T$ (i.e. $x_{t}\not\in S$ for all $t\leq T$), then:

[TABLE]

The main result in [10] shows how constructing such a supermartingale for an optimization algorithm can be used to obtain a bound on the probability that the algorithm has not visited the success region after a certain number of iterations. Under the considered stochastic scheduling model, the authors employ a parameter $\tau$, that denotes the worst-case expected delay caused by the parallel updates. Under the additional assumption that the stochastic gradients contain a single non-zero entry, the following result is derived:

Theorem 6.3** ([10]).**

Consider the SGD algorithm for optimizing a convex function $f$ that satisfies the assumptions above and under the asynchronous model of [10], with a success region $S=\{x\text{ }|\text{ }\|x-x^{*}\|^{2}\leq\epsilon\}$ and with learning rate $\alpha=\frac{c\epsilon\vartheta}{M^{2}+2LM\tau\sqrt{\epsilon}}$ for some constant $\vartheta\in\left(0,1\right)$. Then the probability of the event $F_{T}$ that $x_{i}\not\in S$ for all $i\leq T$ is:

[TABLE]

Again, the bound decreases linearly with the number of iterations. However, there is an additive term that increases with $\tau$. In particular, the bound on the failure probability is worse than in the sequential SGD case described previously.

6.2 Convergence Analysis

We now apply a martingale analysis similar to the one in [10] to obtain results about the rate of convergence of SGD under the Asynchronous Shared Memory model. We denote the maximum delay at time $t$ by $\tau_{t}\geq 0$. We also assume that all $\tau_{t}$ are bounded by some maximum $\tau_{\textrm{max}}$, i.e. $\tau_{t}\leq\tau_{\max}$ for all $t$.

Note that at any time $t$, the gradient is computed based on a view $v_{t}$ that might be missing updates from only the last $\tau_{t}$ iterations. Therefore,

[TABLE]

Now since for any $x\in\mathbb{R}^{d}$, we have $\|x\|_{2}\leq\|x\|_{1}\leq\sqrt{d}\|x\|_{2}$, it follows that:

[TABLE]

For the subsequent analysis we will also need the following:

Lemma 6.4**.**

For any $t$:

[TABLE]

Proof.

By Lemma 6.2, we know that for any constant $K$ and for any $Kn$ consecutive steps $t+1,...,t+Kn$, $\tau_{t+i}>Kn$ for at most $n$ indexes. Hence,

[TABLE]

This holds for any positive $K$. The bound is minimized at $K=\sqrt{\frac{\tau_{\textrm{max}}}{n}}$, which yields the result. ∎

Next, we obtain a bound on the probability that the algorithm has not visited a given success region $S=\{x\text{ }|\text{ }\|x-x^{*}\|^{2}\leq\epsilon\}$. To this end, we show a result similar to the one in Theorem 1 in [10]. We will assume the existence of a rate supermartingale with respect to the underlying sequential SGD process that is Lipschitz in its first coordinate and show that this can be used to obtain a bound on the failure probability. The exact assumptions on $W$ are as follows:

• •

$W$ is a supermartingale with horizon $B$ with respect to the sequential SGD process $x_{t+1}=x_{t}-\alpha\widetilde{g}\left(x_{t}\right)$. Note that $W$ need not be a supermartingale with respect to the lock-free SGD algorithm.

• •

For any $T>0$, if $x_{i}\not\in S$ for all $i\leq T$, then: $W_{T}\left(x_{T},\ldots,x_{0}\right)\geq T.$ Otherwise, we say that the algorithm has succeeded at time $T$.

• •

$W$ is Lipschitz continuous in the current iterate with parameter $H$, i.e. for all $t,u,v$ and any $x_{t-1},...,x_{0}$:

[TABLE]

Under these assumptions, we can prove our main technical claim, whose proof is deferred to the Appendix.

Theorem 6.5**.**

Assume that $W$ is a rate supermartingale with horizon $B$ for the sequential SGD algorithm and that $W$ is $H$-Lipschitz in the first coordinate. Assume further that $\alpha^{2}HLMC\sqrt{d}<1$, where $C=2\sqrt{\tau_{\textrm{max}}n}$. Then for any $T\leq B$, the probability that the lock-free SGD algorithm has not succeeded at time $T$ (that is, the probability of the event $F_{T}$ that $x_{i}\not\in S$ for all $i\leq T$) is:

[TABLE]

We now apply the result with a particular choice for the martingale $W_{t}$. We use the process proposed in [10] in the case of convex optimization, which they show is a rate supermartingale for the sequential SGD process. More precisely, we have the following:

Lemma 6.6** ([10]).**

Define the piecewise logarithm function to be

[TABLE]

Define the process $W_{t}$ by:

[TABLE]

If the algorithm has not succeeded by timestep $t$ (i.e. $x_{i}\not\in S$ for all $i\leq t$) and by $W_{t}=W_{u-1}$ whenever $x_{i}\in S$ for some $i\leq t$ and $u$ is the minimal index with this property. Then $W_{t}$ is a rate supermartingale for sequential SGD with horizon $B=\infty$. It is also $H$-Lipschitz in the first coordinate, with $H=2\sqrt{\epsilon}\left(2\alpha c\epsilon-\alpha^{2}M^{2}\right)^{-1}$, that is for any $t,u,v$ and any sequence $x_{t-1},\ldots,x_{0}$:

[TABLE]

Using this martingale in Theorem 6.5, we obtain the following result, whose proof is deferred to the Appendix:

Corollary 6.7**.**

Assume that we run the lock-free SGD algorithm under the Asynchronous Shared Memory model for minimizing a convex function $f$ satisfying the listed assumptions. Set the learning rate to:

[TABLE]

for some constant $\vartheta\in(0,1]$. Then for any $T>0$ the probability that $x_{i}\not\in S$ for all $i\leq T$ is:

[TABLE]

Choosing $\vartheta=1$ gives the smallest value for the upper bound under this setting. We can impose an arbitrary small learning rate by selecting a small $\vartheta$, while still ensuring convergence.

7 Iterated Algorithm with Guaranteed Convergence

The procedure in Algorithm 1 has the property that, for an appropriately chosen stopping time $T$, it will eventually reach the “success” region, where the distance to the optimal parameter value falls below $\epsilon$. However, due to asynchrony and adversarial updates, threads might perform updates which cause them to leave the success region: a delayed thread might apply stale gradients to the model, overwriting the progress.

We now present an extension of the algorithm which deals with this problem, and allows us to converge to a success region of radius $\epsilon$ around the optimum $x^{*}$, for any $\epsilon>0$. The algorithm will run a series of epochs, each of which is a series of $T$ SGD iterations, executed using the procedure in Algorithm 1. The only difference between epochs is that they are executed with an exponentially decreasing learning rate $\alpha$. (We note that this epoch pattern is already used in many settings, such as neural network training.) The epochs share the model $X$, with the critical note that we require that a gradient update can only be applied to $X$ in the same epoch when it was generated. This condition can be enforced either by maintaining an epoch counter, on which threads condition their update via double-compare-single-swap (DCAS), or by having a distinct model allocated for each epoch.

The only distinct epoch is the last, in which threads each aggregate the gradients they produced locally. At the end of the epoch, the threads will collect all local gradients locally. As we show in the Appendix, the model will be guaranteed to be close to optimal in expectation.

Corollary 7.1**.**

The $\mathsf{FullSGD}$ procedure given in Algorithm 2 guarantees that $\operatorname{\mathbb{E}}[\|r-x^{\star}\|]\leq\epsilon$ after executing for $O(T\log\left(\alpha{2Mn/\sqrt{\epsilon}}\right))$ iterations.

We can characterize the probability of success by reducing the target $\epsilon$, and applying Markov’s inequality.

8 Discussion

Learning Rates versus Asynchronous Convergence.

An immediate consequence of our lower bound in Theorem 5.1 is that the algorithm must either have a low initial learning rate, or lower the learning rate across multiple iterations (as in Algorithm 2) in order to be able to withstand adversarial delays. Otherwise, the adversary can always apply stale gradients generated at a far enough time in the past to nullify progress. An alternative approach, which we did not consider here, would be to introduce a “momentum” term by which the current model value is multiplied [20].

Lower Bounds versus Upper Bounds.

The attentive reader might find it curious that our lower bound suggests a linear slowdown in $\tau_{\max}$, whereas our upper bound suggests that the slowdown is linear in $O(\sqrt{\tau_{\max}{}n})$. However, a close examination of the preconditions for these two results will reveal that they are in fact complementary: in the lower bound, given fixed learning rate $\alpha$, the adversary needs to set a large delay $\tau_{\max}{}\geq(\log(\alpha/2))/\log(1-\alpha)$ in order to slow down convergence. At the same time, the upper bound in Theorem 6.5 requires that $2\alpha^{2}HLM\sqrt{d\tau_{\max}n}<1$, which is incompatible with the above condition. Specifically, our improved convergence bound shows that asynchronous SGD converges faster and for a wider range of parameters than previously known.

Why is Asynchronous SGD Fast in Practice?

In a nutshell, Theorem 6.5 shows that the gap in convergence between asynchronous SGD and the sequential variant becomes negligible if $\alpha^{2}HLMC\sqrt{d\tau_{\max}n}\ll 1$. Intuitively, this condition holds in practice since gradients are often sparse, meaning that $d$ is low, the delay factors $\tau_{\max}$ and $\tau_{avg}$ are not set adversarially, and the learning rate $\alpha$ can be set by the user to be small enough to offset any increase in the other terms. In particular, $\tau_{\max}$ is limited by the staleness of updates in the write buffer at each core, which is well bounded in practice [21].

At the same time, it is important to note that, while we “sequentialize” iterations in our analysis, up to $n$ iterations may happen in parallel at any time, reducing the wall-clock convergence time by up to a factor of $n$. Thus, the practical trade-off is between any slow-down caused by asynchrony, and the parallelism due to multiple computation threads.

9 Acknowledgements

We would like to thank Martin Jaggi for useful discussions. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 665385.

Appendix A Deferred Proofs

A.1 Proof of Theorem 6.5

**Theorem 6.5 **.

Assume that $W$ is a rate supermartingale with horizon $B$ for the sequential SGD algorithm and that $W$ is $H$-Lipschitz in the first coordinate. Assume further that $\alpha^{2}HLMC\sqrt{d}<1$, where $C=2\sqrt{\tau_{\textrm{max}}n}$. Then for any $T\leq B$, the probability that the lock-free SGD algorithm has not succeeded at time $T$ (that is, $x_{i}\not\in S$ for all $i\leq T$) is:

[TABLE]

Proof.

Consider a process defined by $V_{0}=W_{0}$ and by:

[TABLE]

whenever $x_{i}\not\in S$ for all $0\leq i\leq t$. Finally, if $x_{i}\in S$ for some $0\leq i\leq t$, then define

[TABLE]

where $u$ is the minimal index, such that $x_{u}\in S$. Assume that the algorithm has not succeeded at time $t$. Using the Lipschitzness of $W$:

[TABLE]

Now we take expectation with respect to the gradient at time $t$ and condition on the past (denoted by $\mathbb{E}_{t|.}$). Using the Lipschitzness of $\widetilde{g}$ and the supermartingale property of $W_{t}$ for the first step and the bound on the expected norm of the gradient and Lemma 6.4 for the second:

[TABLE]

This inequality also trivially holds in the case when the algorithm has succeeded at time $t$. Hence, the process $V_{t}$ is a supermartingale for the lock-free SGD process.

Note also that if the algorithm has not succeeded at time $T$, then $W_{T}\geq T$ and hence:

[TABLE]

It follows that $V_{t}\geq 0$ for all $t$. We also have that $V_{0}=W_{0}$. Now for any $T>0$:

[TABLE]

We conclude that:

[TABLE]

∎

A.2 Proof of Corollary 6.7

**Corollary 6.7 **.

Assume that we run the lock-free SGD algorithm under the Asynchronous Shared Memory model for minimizing a convex function $f$ satisfying the listed assumptions. Set the learning rate to:

[TABLE]

for some constant $\vartheta\in(0,1]$. Then for any $T>0$ the probability that $x_{i}\not\in S$ for all $i\leq T$ is:

[TABLE]

Proof.

Substituting and using the result from [10] that

[TABLE]

we obtain that:

[TABLE]

Substituting the suggested value for the learning rate:

[TABLE]

∎

A.3 Analysis of Algorithm 2

**Corollary 7.1 **.

The $\mathsf{FullSGD}$ procedure given in Algorithm 2 guarantees that $\operatorname{\mathbb{E}}[\|r-x^{\star}\|]\leq\epsilon$ after executing for $O(T\log\left(\alpha{2Mn/\sqrt{\epsilon}}\right))$ iterations.

Proof Sketch.

The main idea behind the analysis is as follows. By Theorem 6.5, we know that, with high probability, there exists a time $t$ in each epoch where the aggregated gradients $x_{t}$ enter the success region, i.e. $\|x_{t}-x^{*}\|^{2}\leq\epsilon$, for a fixed parameter $\epsilon$. The algorithm will guarantee that the model will not leave the success region before the end of the last epoch. This is ensured via our choice of the learning rate.

Let us now focus on the last epoch. Fix an $\epsilon>0$; we wish to prove that, $\|x_{T}-x^{\star}\|^{2}\leq\epsilon$ at the end of this epoch, in expectation. We fix the success condition of $\mathsf{EpochSGD}$ such that that there exists an iteration $t$ in the epoch such that $\|x_{t}-x^{\star}\|\leq\sqrt{\epsilon}/2$. We note that the adversary may attempt to schedule “stale” updates, generated earlier in the execution, to cause the algorithm to leave the success region. However, we notice that there can be at most $n-1$ gradients generated before time $t$, which have not been applied yet. Denote these gradients by $(G(v_{\theta_{i}}))_{i=1,\ldots,n-1}$.

Finally, we claim that, in expectation, the distance between the final model $x_{T}$ and the optimum is upper bounded by

[TABLE]

∎