Coded convolution for parallel and distributed computing within a deadline

TL;DR

This paper introduces a coding-based approach to distributed convolution that enhances resilience against slow or faulty processors, significantly increasing the probability of completing computations within strict deadlines.

Contribution

It develops a novel asymptotic failure exponent analysis for coded distributed computing, providing closed-form expressions and improved deadline reliability over traditional methods.

Findings

Coding improves resilience against stragglers in distributed convolution.

The method dramatically increases the probability of meeting deadline constraints.

Closed-form expressions are derived for various computation time models.

Abstract

We consider the problem of computing the convolution of two long vectors using parallel processing units in the presence of "stragglers". Stragglers refer to the small fraction of faulty or slow processors that delays the entire computation in time-critical distributed systems. We first show that splitting the vectors into smaller pieces and using a linear code to encode these pieces provides better resilience against stragglers than replication-based schemes under a simple, worst-case straggler analysis. We then demonstrate that under commonly used models of computation time, coding can dramatically improve the probability of finishing the computation within a target "deadline" time. As opposed to the more commonly used technique of expected computation time analysis, we quantify the exponents of the probability of failure in the limit of large deadlines. Our exponent metric captures the probability of failing to finish before a specified deadline time, i.e. , the behavior of the "tail". Moreover, our technique also allows for simple closed form expressions for more general models of computation time, e.g. shifted Weibull models instead of only shifted exponentials. Thus, through this problem of coded convolution, we establish the utility of a novel asymptotic failure exponent analysis for distributed systems.

Figures

Equations

$$\underset{s_1,s_2}{\min }(s_1+s_2)\log (s_1+s_2)$$

$$\text{subject to }{s}_1{s}_2=N_1{N}_2/P$$

$$1\le s_1\le N_1,1\le s_2\le N_2$$

$$K=\Big{(}\frac{P}{r}-1\Big{)}r+1=P-\frac{Ps^{2}}{N_{1}N_{2}}+1$$

$${\mathbf{G}}^T\left[\begin{array}{c}{\mathbf{a}}_1\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\end{array}\right]=\left[\begin{array}{cccc}1& g_1& \dots & g_1^{\frac{N_1}{s}-1}\\ 1& ⋮& \ddots & ⋮\\ 1& g_{\frac{Ps}{N_2}}& \dots & g_{\frac{Ps}{N_2}}^{\frac{N_1}{s}-1}\end{array}\right]\left[\begin{array}{c}{\mathbf{a}}_1\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\end{array}\right]=\left[\begin{array}{c}{\tilde{\mathbf{a}}}_1\\ ⋮\\ {\tilde{\mathbf{a}}}_{\frac{Ps}{N_2}}\end{array}\right]$$

$${\mathbf{G}}_L^T\left[\begin{array}{c}{\mathbf{a}}_1\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\end{array}\right]=\left[\begin{array}{c}{\tilde{\mathbf{a}}}_{l_1}\\ ⋮\\ {\tilde{\mathbf{a}}}_{l_{\frac{N_1}{s}}}\end{array}\right].$$

$$\left[\begin{array}{c}{\mathbf{a}}_1\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\end{array}\right]=[{\mathbf{G}}_L^T{]}^{-1}\left[\begin{array}{c}{\tilde{\mathbf{a}}}_{l_1}\\ ⋮\\ {\tilde{\mathbf{a}}}_{l_{\frac{N_1}{s}}}\end{array}\right]=\mathbf{B}\left[\begin{array}{c}{\tilde{\mathbf{a}}}_{l_1}\\ ⋮\\ {\tilde{\mathbf{a}}}_{l_{\frac{N_1}{s}}}\end{array}\right]$$

$${\mathbf{a}}_i\ast {\mathbf{x}}_j={\mathbf{B}}_i\left[\begin{array}{c}{\tilde{\mathbf{a}}}_{l_1}\ast {\mathbf{x}}_j\\ ⋮\\ {\tilde{\mathbf{a}}}_{l_{\frac{N_1}{s}}}\ast {\mathbf{x}}_j\end{array}\right]\Rightarrow \left[\begin{array}{c}{\mathbf{a}}_1\ast {\mathbf{x}}_j\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\ast {\mathbf{x}}_j\end{array}\right]=\mathbf{B}\left[\begin{array}{c}{\tilde{\mathbf{a}}}_{l_1}\ast {\mathbf{x}}_j\\ ⋮\\ {\tilde{\mathbf{a}}}_{l_{\frac{N_1}{s}}}\ast {\mathbf{x}}_j\end{array}\right]$$

$$K=\Big{(}\frac{N_{2}}{s}-1\Big{)}\frac{Ps}{N_{2}}+\frac{N_{1}}{s}=P-\frac{Ps}{N_{2}}+\frac{N_{1}}{s}$$

$$K_{Coded}=P-\frac{Ps}{N_2}+\frac{N_1}{s}<P-\frac{P{s}^2}{N_1{N}_2}+1=K_{replication}$$

$$F_{s}(t)=1-e^{-\big{(}\mu\left(\frac{t}{2Cs\log{(2s)}}-1\right)\big{)}^{\alpha}}\ \ \forall\ t\ \geq 2Cs\log{(2s)}$$

$$P_s^f(t)$$

$$\displaystyle=\sum_{i=0}^{K-1}\binom{P}{i}\Big{(}\frac{(F_{s}(t))}{1-F_{s}(t)}\Big{)}^{i}(1-F_{s}(t))^{P}$$

$$\displaystyle=\Theta\Big{(}c(P,K)\Big{(}\frac{F_{s}(t)}{1-F_{s}(t)}\Big{)}^{K-1}(1-F_{s}(t))^{P}\Big{)}$$

$$\displaystyle\binom{P}{K-1}\Big{(}\frac{(F_{s}(t))}{1-F_{s}(t)}\Big{)}^{K-1}(1-F_{s}(t))^{P}\leq\sum_{i=0}^{K-1}\binom{P}{i}\Big{(}\frac{(F_{s}(t))}{1-F_{s}(t)}\Big{)}^{i}(1-F_{s}(t))^{P}\leq\sum_{i=0}^{K-1}\binom{P}{i}\Big{(}\frac{(F_{s}(t))}{1-F_{s}(t)}\Big{)}^{K-1}(1-F_{s}(t))^{P}$$

$$c_{1}(P,K)\Big{(}\frac{(F_{s}(t))}{1-F_{s}(t)}\Big{)}^{K-1}(1-F_{s}(t))^{P}\leq\sum_{i=0}^{K-1}\binom{P}{i}\Big{(}\frac{(F_{s}(t))}{1-F_{s}(t)}\Big{)}^{i}(1-F_{s}(t))^{P}\leq c_{2}(P,K)\Big{(}\frac{(F_{s}(t))}{1-F_{s}(t)}\Big{)}^{K-1}(1-F_{s}(t))^{P}$$

$$\underset{t\to \infty }{\lim }\frac{\log (P_s^f(t))}{t^{\alpha }}$$

$$=\underset{t\to \infty }{\lim }(K-1)\frac{\log \left(e^{{\mu }^{\alpha }{\left(\frac{t}{(2Cs)\log (2s)}-1\right)}^{\alpha }}-1\right)}{t^{\alpha }}+P\frac{\log \left(e^{-{\mu }^{\alpha }{\left(\frac{t}{(2Cs)\log (2s)}-1\right)}^{\alpha }}\right)}{t^{\alpha }}$$

$$\to -\frac{(P-K+1){\mu }^{\alpha }}{((2Cs)\log (2s){)}^{\alpha }}$$

$$\displaystyle=-\frac{\big{(}\frac{Ps}{N_{2}}-\frac{N_{1}}{s}+1\big{)}\mu^{\alpha}}{((2Cs)\log{(2s)})^{\alpha}}$$

$$\epsilon(s)=-\frac{\big{(}\frac{Ps}{N_{2}}-\frac{N_{1}}{s}+1\big{)}\mu^{\alpha}}{((2Cs)\log{(2s)})^{\alpha}}$$

$$\epsilon(s)|_{s=\sqrt{N_{1}N_{2}/P}}=-\frac{\mu^{\alpha}}{\Big{(}2C\sqrt{\frac{N_{1}N_{2}}{P}}\log{2\sqrt{\frac{N_{1}N_{2}}{P}}}\Big{)}^{\alpha}}$$

$$\frac{\big{|}\epsilon\left(2\sqrt{N_{1}N_{2}/P}\right)\big{|}}{\left|\epsilon\left(\sqrt{N_{1}N_{2}/P}\right)\right|}=\frac{\frac{3}{2}\sqrt{\frac{PN_{1}}{N_{2}}}+1}{2^{\alpha}\Big{(}1+\frac{\log{2}}{\log{2\sqrt{N_{1}N_{2}/P}}}\Big{)}^{\alpha}}>\frac{3.\sqrt{\frac{PN_{1}}{N_{2}}}}{2.4^{\alpha}}$$

$$\underset{s_1,s_2}{\min }C(2s_1)(\log (2s_2)+1)$$

$$\text{subject to }{s}_1{s}_2=\frac{N_1{N}_2}{P}$$

$$1\le s_1\le N_1,1\le s_2\le N_2$$

$$f(s_1)=C(2s_1)\left(\log \left(\frac{2N_1{N}_2}{P{s}_1}\right)+1\right)$$

$$f^{\prime }(s_1)=2C\left(\log \frac{2N_1{N}_2}{P}-\log s_1\right)$$

$${\mathbf{G}}^T\left[\begin{array}{c}{\mathbf{a}}_1\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\end{array}\right]=\left[\begin{array}{cccc}1& g_1& \dots & g_1^{\frac{N_1}{s}-1}\\ 1& ⋮& \ddots & ⋮\\ 1& g_{\frac{Ps}{N_2}}& \dots & g_{\frac{Ps}{N_2}}^{\frac{N_1}{s}-1}\end{array}\right]\left[\begin{array}{c}{\mathbf{a}}_1\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\end{array}\right]=\left[\begin{array}{c}{\tilde{\mathbf{a}}}_1\\ ⋮\\ {\tilde{\mathbf{a}}}_{\frac{Ps}{N_2}}\end{array}\right]$$

$$(s)\left(\frac{Ps}{N_2}\right){\left(\log \frac{Ps}{N_2}\right)}^2<sP(\log P{)}^2$$

$$\left[\begin{array}{c}{\mathbf{a}}_1\ast {\mathbf{x}}_j\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\ast {\mathbf{x}}_j\end{array}\right]=[{\mathbf{G}}_L^T{]}^{-1}\left[\begin{array}{c}{\tilde{\mathbf{a}}}_{l_1}\ast {\mathbf{x}}_j\\ ⋮\\ {\tilde{\mathbf{a}}}_{l_{\frac{N_1}{s}}}\ast {\mathbf{x}}_j\end{array}\right]$$

$$\left[\begin{array}{c}{\mathbf{a}}_1\ast {\mathbf{x}}_j\\ ⋮\\ {\mathbf{a}}_{\frac{N_1}{s}}\ast {\mathbf{x}}_j\end{array}\right]={\left[\begin{array}{cccc}1& g_{l_1}& \dots & g_{l_1}^{\frac{N_1}{s}-1}\\ 1& g_{l_2}& \dots & g_{l_2}^{\frac{N_1}{s}-1}\\ & ⋮& \ddots & ⋮\\ 1& g_{l_{\frac{N_1}{s}}}& \dots & g_{l_{\frac{N_1}{s}}}^{\frac{N_1}{s}-1}\end{array}\right]}^{-1}\left[\begin{array}{c}{\tilde{\mathbf{a}}}_{l_1}\ast {\mathbf{x}}_j\\ ⋮\\ {\tilde{\mathbf{a}}}_{l_{\frac{N_1}{s}}}\ast {\mathbf{x}}_j\end{array}\right]$$

$$(2s)\left(\frac{N_1{N}_2}{s^2}\right){\left(\log \frac{N_1}{s}\right)}^2<2sP(\log P{)}^2$$

$$(2s)\frac{N_1}{s}\frac{N_2}{s}\le 2sP$$

$$\text{Enc.}+\text{Dec.}+\text{Add.}$$

Full text

I Introduction

The operation of convolution has widespread applications in mathematics, physics, statistics and signal processing. Convolution plays a key role in solving inhomogeneous differential equations to determine the response of a system under initial conditions [1]. Convolution of any arbitrary input function with the pre-specified impulse response of the system, also known as the Green’s function [1], provides the response of the system to any arbitrary input function. Convolution is also useful for filtering or extraction of features, in particular for Convolutional Neural Networks (CNNs), which are frequently used in time-critical systems like autonomous vehicles. In [2], Dally points out that application of machine learning tools in low-latency inference problems is primarily bottlenecked by the computation time.

In this paper, we propose a novel coded convolution for time-critical distributed systems prone to straggling and delays, for fast and reliable computation within a target deadline. We also propose a novel analysis technique based on a new metric – the exponent of the asymptotic probability of failure to meet a target deadline – for comparison of various delay tolerant techniques in distributed systems. Our contributions are two fold:-

We go beyond distributed matrix-vector products, and explore the problem of distributed convolution (building on [3]) in a straggler prone time-critical distributed scenario. We propose a novel strategy of splitting the vectors to be convolved and coding them, so as to perform fast and reliable convolution within specified deadlines. Moreover our strategy also allows for encoding and decoding online as compared to existing work on coded matrix-vector products which require encoding to be performed as a pre-processing step because of its high computational complexity. 2. 2.

We introduce a new analysis technique to compare the performance of various fault/delay tolerant strategies in distributed systems. Under a more generalized shifted Weibull[4] computation time model (that also encompasses shifted exponential), we demonstrate the utility of the proposed deadline-driven analysis technique for the coded convolution problem.

The motivation for our work arises from the increasing drive towards distributed and parallel computing with ever-increasing data dimensions. Parallelization reduces the per processor computation time, as the task to be performed at each processor is substantially reduced. However, the benefits of parallelization are often limited by “stragglers”: a few slow processors that delay the entire computation. These processors can cause the probabilistic distribution of computation times to have a long tail, as pointed out in the influential paper of Dean and Barroso[5]. For time-critical applications, stragglers hinder the completion of tasks within a specified deadline, since one has to wait for all the processors to finish their individual computations.

The use of replication[6], [7], [8] is a natural strategy for dealing with stragglers in distributed systems since the computation only requires a subset of the processors to finish. Error correcting codes have been found to offer advantages over replication in various situations. The use of simple check-sum based codes for correcting errors in linear transforms dates back to the ideas of algorithmic fault tolerance [9] and its extensions in [4] [10]. A strategy of coding multiple parallel convolutions over reals for error tolerance is proposed in [3]. An alternative technique for error-resilient convolutions, that limits itself to finite-field vectors and requires coding both vectors into residue polynomials, was proposed in [11]. Recently, the use of erasure codes, e.g., MDS code based techniques [12][13], has been explored for speeding up matrix-vector products in distributed systems. In [14], we introduce a novel class of codes called – Short-Dot codes – that compute multiple straggler-tolerant short dot products towards computing a large matrix-vector product. The most common metric for comparing the performance of various strategies in existing literature is expected computation time that often uses a shifted exponential computation time model. However, this method of analysis becomes unwieldy when extended to higher order moments. Moreover, expected time does not capture the probability of failure to meet a specific deadline which could be important in time-critical applications.

II Assumptions and Problem Formulation

Before proceeding further, note that the computational complexity of convolving two vectors of any lengths, say $m_{1}$ and $m_{2}$ using Fast Fourier Transform (FFT) is $\Theta\left((m_{1}+m_{2}-1)(\log{(m_{1}+m_{2}-1)}+1)\right)$ [15] [16]. When one of $m_{1}$ or $m_{2}$ is much smaller than the other, say $\log{m_{1}}\ll\log{m_{2}}$, then the computational complexity can be reduced even further using overlap methods like overlap add [17]. When using overlap methods, the longer vector is divided into smaller pieces of length comparable to the smaller one and each piece is convolved separately. The outputs are then combined together. In this paper, for conceptual simplicity we make some assumptions on the computational complexity of convolution, under different scenarios.

Scenario 1: We assume that when $m_{1}$ and $m_{2}$ are comparable, the computational complexity of convolution using FFT is given by $C(m_{1}+m_{2})(\log{(m_{1}+m_{2})})$ where $C$ is a constant.

Scenario 2: We assume that when the length of one vector is sufficiently smaller than another, specifically $\log(m_{2})=o(\log(m_{1}))$, then the computational complexity using overlap methods[17] is $2Cm_{1}(\log{(2m_{2})}+1)$ where $C$ is a constant.

Problem Formulation

We consider the problem of convolution of a vector $\bm{a}$ of length $N_{1}$ with an unknown input vector $\bm{x}$ of length $N_{2}$, using $P$ parallel processing units. The computation goal is to convolve the two vectors reliably in a distributed and parallelized fashion, in the presence of stragglers, so as to prevent failure to meet a specified deadline.

Assumption: For the ease of theoretical analysis, we assume that $2\sqrt{\frac{N_{1}N_{2}}{P}}\leq\min\{N_{1},N_{2}\}$. This ensures that $N_{1}$ and $N_{2}$ are not too far from each other for relatively small $P$.

Note that, the operation of convolution can be represented as the multiplication of a Toeplitz matrix, with a vector, requiring a naive computational complexity of $\Theta(N_{1}N_{2})$. One might wonder if we should parallelize this matrix vector product and, in doing so, use techniques from [12][14] that make matrix-vector products resilient to straggling. However, using such a parallelization scheme over the $P$ given processors, the computational complexity per processor cannot be reduced to below $\Theta(\frac{N_{1}N_{2}}{P})$. If $P$ is not large enough, this is still substantially larger than $C(N_{1}+N_{2})\log{(N_{1}+N_{2})}$, the computational complexity of convolution using FFT on a single processor under Scenario 1.

III The Naive Uncoded Strategy

An uncoded strategy to perform the convolution is to divide both the vectors $\bm{a}$ (length $N_{1}$) and $\bm{x}$ (length $N_{2}$) into smaller parts of lengths $s_{1}$ and $s_{2}$ respectively. Let the parts of the vector $\bm{a}$ be denoted by $\bm{a}_{1},\bm{a}_{2},\dots,\bm{a}_{\frac{N_{1}}{s_{1}}}$, each of length $s_{1}$. Similarly, the parts of vector $\bm{x}$ are denoted by $\bm{x}_{1},\bm{x}_{2},\dots,\bm{x}_{\frac{N_{2}}{s_{2}}}$. We first consider the case where the entire convolution is computed in parallel using all the given $P$ processors, in one shot. Consider the following algorithm:

For $i=1,2,\dots,N_{1}/s_{1}$

                                                       For $j=1,2,\dots,N_{2}/s_{2}$

                                                               Convolve $\bm{a}_{i}\ast\bm{x}_{j}$ in parallel

Thus, we must have $\frac{N_{1}}{s_{1}}\frac{N_{2}}{s_{2}}=P$. Depending on the lengths $s_{1}$ and $s_{2}$, the convolutions can be performed under Scenario 1 or 2. First consider the case where we perform the convolutions under Scenario 1, choosing $s_{1}$ and $s_{2}$ of comparable lengths. Then, for minimizing per processor computational complexity, we have the following optimization problem:

[TABLE]

A quick differentiation reveals that the above minimization is achieved when $s_{1}=s_{2}=\sqrt{N_{1}N_{2}/P}$ (say $s$). Thus, the computational complexity required for convolution under Scenario 1 would be C\big{(}2\sqrt{\frac{N_{1}N_{2}}{P}}\big{)}\log{\Big{(}2\sqrt{\frac{N_{1}N_{2}}{P}}\Big{)}}. We show in Appendix A that operating under Scenario 2 requires equal or higher per processor complexity than the aforementioned strategy. We also show that, when we use the $P$ processors several times instead of operating in a single shot, the computational complexity only increases.

Thus, for uncoded convolution, we provably show that the minimum computational complexity per processor in order sense is attained when $s_{1}=s_{2}=\sqrt{N_{1}N_{2}/P}$ (say $s=\sqrt{N_{1}N_{2}/P}$). Now, we can perform $P$ convolutions of $s$-length vectors, convolving each of the $\frac{N_{1}}{s}$ parts of $\bm{a}$ with $\frac{N_{2}}{s}$ parts of $\bm{x}$ in parallel. A convolution of length $s=\sqrt{N_{1}N_{2}/P}$ in each processor has a computational complexity of $2C\sqrt{\frac{N_{1}N_{2}}{P}}\log{2\sqrt{\frac{N_{1}N_{2}}{P}}}$, where $C$ is a constant which is independent of the length of the convolution. Fig. 1 shows the uncoded convolution strategy.

Reconstruction: First let us discuss how to reconstruct each $\bm{a}\ast\bm{x}_{j}$ of length $N_{1}+s_{2}$ from successfully computed convolution outputs $\{\bm{a}_{i}\ast\bm{x}_{j}|i=1,2,\dots,N_{1}/s_{1}\}$ each of size $s_{1}+s_{2}$. Consider the following algorithm:

**For ** $i=1,2,\dots,N_{1}/s_{1}$

                                                      Pad $\bm{a}_{i}\ast\bm{x}_{j}$ with [math]s to the right to get length $N_{1}+s_{2}$

                                                      Shift $\bm{a}_{i}\ast\bm{x}_{j}$ ** right by ** $(i-1)s_{1}$

                                             Finally add all ** $N_{1}/s_{1}$ ** shifted outputs

The algorithm is also illustrated in Fig. 2 for $s_{1}=s_{2}=s$. Now, we may similarly obtain $\bm{a}\ast\bm{x}$, from $\{\bm{a}\ast\bm{x}_{j}|j=1,2,\dots,N_{2}/s_{2}\}$, by shifting each result right by $(j-1)s_{2}$, and adding the shifted results all together.

Note that this uncoded strategy requires all the $P$ processors to finish. If some processors straggle, the entire computation is delayed, and may fail to finish within the specified deadline. In the next sections, we first analyze a replication strategy that introduces redundancy and does not require all the processors to finish, and then propose our coded convolution strategy.

IV The Replication Strategy

In the previous section, we have shown that the strategy of minimizing the per processor computational complexity in convolution is to divide each vector into equal sized portions, and convolve in parallel processors, all in one shot. Now, let us consider a $(P,r)$ replication strategy, where every sub task has $r$ replicas or copies. We basically divide the task of convolution into $P/r$ equal parts, and each part has $r$ replicas, so as to use all the given $P$ processors. Then, the length of the vectors to be convolved at each processor is given by $s=\sqrt{\frac{N_{1}N_{2}r}{P}}$.

Theorem 1

For the convolution of a vector of length $N_{1}$ with another vector of length $N_{2}$ using $P$ processors, a $(P,r)$ replication strategy needs $K=P-\frac{Ps^{2}}{N_{1}N_{2}}+1$ short convolutions of length $s=\sqrt{\frac{N_{1}N_{2}r}{P}}$ to finish in the worst case.

Proof:

Observe that there are $\frac{P}{r}$ tasks that need to finish, each having $r$ replicas. The worst case wait arises when the replicas of all but one of the $\frac{P}{r}$ tasks have completed before any one copy of the last task has finished. Thus,

[TABLE]

This proves the theorem.∎

V The Coded Convolution Strategy

We now describe our proposed coded convolution strategy which ensures that computation outputs from a subset of the $P$ processors are sufficient to perform the overall convolution.

V-A An $(N_{1},N_{2},P,s)$ Coded Convolution

Let us choose a length $s$ such that $\sqrt{\frac{N_{1}N_{2}}{P}}<s$. Now, we can divide both the vectors $\bm{a}$ and $\bm{x}$ into parts of length $s$, as shown in Fig. 3. The vector $\bm{x}$ has $\frac{N_{2}}{s}$ parts of length $s$, namely $\bm{x}_{1},\dots,\bm{x}_{\frac{N_{2}}{s}}$. Similar to uncoded strategy, for every convolution $\bm{a}\ast\bm{x}_{j}$ we assign $Ps/N_{2}$ processors . The $\frac{N_{1}}{s}$ parts of the vector $\bm{a}$ are encoded using a \big{(}\frac{Ps}{N_{2}},\frac{N_{1}}{s}\big{)} MDS code to produce $\frac{Ps}{N_{2}}$ vectors of length $s$, as discussed below:

Let \bm{G}_{\big{(}\frac{N_{1}}{s}\times\frac{Ps}{N_{2}}\big{)}} be the generator matrix of a $\left(\frac{Ps}{N_{2}},\frac{N_{1}}{s}\right)$ MDS code on the real field, i.e., a matrix such that every $\frac{N_{1}}{s}\times\frac{N_{1}}{s}$ square sub-matrix is invertible [14]. For example, an \big{(}\frac{N_{1}}{s}\times\frac{Ps}{N_{2}}\big{)} Vandermonde matrix satisfies this property. Now we encode the $\bm{a}_{i}$s as follows:

[TABLE]

Here $\{g_{1},g_{2},\dots,g_{\frac{Ps}{N_{2}}}\}$ are all distinct reals. Since every $\frac{N_{1}}{s}\times\frac{N_{1}}{s}$ square sub-matrix of $\bm{G}$ is invertible, any $\frac{N_{1}}{s}$ vectors out of the $\frac{Ps}{N_{2}}$ coded vectors $\{\bm{\tilde{a}}_{1},\dots,\bm{\tilde{a}}_{\frac{Ps}{N_{2}}}\}$ can be linearly combined to generate the original $\frac{N_{1}}{s}$ parts of $\bm{a}$, i.e., $\{\bm{a}_{1},\dots,\bm{a}_{\frac{N_{1}}{s}}\}$. To see this, let $L=\{l_{1},l_{2},\dots,l_{\frac{N_{1}}{s}}\}$ denote a subset of any $\frac{N_{1}}{s}$ distinct indices of $\{1,2,\dots,\frac{Ps}{N_{2}}\}$. Also, let $\{\bm{\tilde{a}}_{l_{1}},\dots,\bm{\tilde{a}}_{l_{\frac{N_{1}}{s}}}\}$ be the set of $\frac{N_{1}}{s}$ coded vectors corresponding to the indices in $L$. Thus,

[TABLE]

Here $\bm{G}^{T}_{L}$ denotes a $\frac{N_{1}}{s}\times\frac{N_{1}}{s}$ square sub-matrix of $\bm{G}^{T}$, consisting of the rows indexed in $L$. Since $\bm{G}^{T}_{L}$ is invertible, we have

[TABLE]

Here, $\bm{B}$ denotes $[\bm{G}^{T}_{L}]^{-1}$. Now, each of these coded vectors $\{\bm{\tilde{a}}_{1},\dots,\bm{\tilde{a}}_{\frac{Ps}{N_{2}}}\}$ is convolved with each part of $\bm{x}$, i.e., $\{\bm{x}_{1},\dots,\bm{x}_{\frac{N_{2}}{s}}\}$ in $P$ separate processors. Fig. 3 shows the coded convolution strategy.

From (4), for any $j$, all the original $\{\bm{a}_{i}\ast\bm{x}_{j}|i=1,2,\dots,\frac{N_{1}}{s}\}$ can be decoded from a linear combination of any $\frac{N_{1}}{s}$ of $\{\bm{\tilde{a}}_{1}\ast\bm{x}_{j},\bm{\tilde{a}}_{2}\ast\bm{x}_{j},\dots,\bm{\tilde{a}}_{\frac{Ps}{N_{1}}}\ast\bm{x}_{j}\}$, in particular, those that finish first. Let $\bm{B}_{i}$ denote a row vector corresponding to the $i^{th}$ row of $\bm{B}$. Thus for any $(i,j)$,

[TABLE]

The decoded outputs can be combined similar to the uncoded strategy. Now, we state some results for Coded Convolution.

Theorem 2

For convolution of a vector of length $N_{1}$ with another vector of length $N_{2}$ using $P$ processors, an $(N_{1},N_{2},P,s)$ Coded Convolution decoder needs $K=P-\frac{Ps}{N_{2}}+\frac{N_{1}}{s}$ short convolutions of length $s=\sqrt{\frac{N_{1}N_{2}}{P}}$ to finish in the worst case.

Proof:

Note that each part of $\bm{x}$ of length $s$ is convolved with $\frac{Ps}{N_{2}}$ vectors, so that any $\frac{N_{1}}{s}$ are sufficient. Now, in the worst case, all but one part of $\bm{x}$ finishes convolution with all $\frac{Ps}{N_{2}}$ vectors before the last one finishes with $\frac{N_{1}}{s}$ vectors. Thus, the number of processors required is at most,

[TABLE]

This proves the theorem. ∎

V-B Comparison with replication strategy:

We first show that the worst-case number of processors required using Coded Convolution is less than that required using replication for the same per-processor computational complexity.

Corollary 1

For convolution of vector of length $N_{1}$ with an unknown input vector of length $N_{2}$ using $P$ processors, an $(N_{1},N_{2},P,s)$ Coded Convolution decoder waits for fewer processors in the worst case as compared to that of a $(P,r)$ replication strategy with equal per processor computational complexity.

Proof:

Recall from Section IV that in a $(P,r)$ replication strategy, the length of the vectors convolved at each processor is given by $\sqrt{\frac{N_{1}N_{2}r}{P}}$. For equal per processor task, this length should be equal to $s$ in Coded Convolution. Using the fact that $s=\sqrt{\frac{N_{1}N_{2}r}{P}}>\sqrt{\frac{N_{1}N_{2}}{P}}$, we get,

[TABLE]

This proves the theorem. ∎

Comment: To justify our focus on the worst case $K$, we also discuss in Section VI how the exponent of the probability of failure to meet a deadline in the asymptotic regime depends on the worst case $K$.

V-C Computational Complexity of Encoding and Reconstruction:

Observe that, the computational complexity of each processor is $(2Cs)\log{(2s)}$ from Scenario 1. To be able to encode and reconstruct online, it is desirable that the encoding complexity and reconstruction complexity, i.e., the total complexity of decoding and subsequent additions are both negligible compared to the per processor complexity, since otherwise the encoder/decoder becomes the primary bottleneck during successful completion of tasks within a specified deadline.

Theorem 3

For an $(N_{1},N_{2},P,s)$ Coded Convolution using a Vandermonde Encoding Matrix, the ratio of the total encoding complexity and reconstruction complexity to the per processor complexity tends to [math], as $N_{1},N_{2},P\to\infty$, when P(\log{P})^{2}=o\Big{(}\log{\sqrt{\frac{N_{1}N_{2}}{P}}}\Big{)}.

A proof based on [18], [19] is provided in Appendix B.

VI Asymptotic Analysis of Exponents

We now perform an asymptotic analysis of the exponent of the probability of failure to meet a deadline in the limit of the deadline diverging to infinity. Let $F_{s}(t)$ denote the probability that a convolution of two vectors of length $s$ is computed in a single processor within time $t$, based on a shifted Weibull model. Thus

[TABLE]

Here $\mu>0$ is a straggling parameter, $\alpha>0$ denotes the exponent of the delay distribution and $C$ denotes the constant of convolution that is independent of the length of the vectors. For shifted exponential model, $\alpha=1$.

Let $P^{f}_{s}(t)$ be the probability of failure to finish the convolution of two vectors of length $s$ at or before time $t$.

From Theorem 3, when P(\log{P})^{2}=o\big{(}\log{\sqrt{N_{1}N_{2}/P}}\big{)}, the decoding complexity is negligible compared to the per processor complexity. Thus, only the straggling in the parallel processors is the bottleneck for failure to meet a deadline. We now compute the failure-exponent for the coded strategy, in the limit of $t\to\infty$. For the coded strategy, at most $K=P-\frac{Ps}{N_{2}}+\frac{N_{1}}{s}$ processors need to finish computation in the worst case. An upper bound on the failure probability is thus given by the probability that at most $K-1$ processors have finished the convolution of two $s$ length vectors. Thus,

[TABLE]

Here $c(P,K)$ denotes a function of $P$ and $K$, that is independent of $t$. The last line follows since for $t$ large enough, $\frac{(F_{s}(t))}{(1-F_{s}(t))}\gg 1$, and thus for the purpose of analysis of the failure exponent for large $t$, the binomial summation is of the same order as the largest term, as we show here:

[TABLE]

This implies,

[TABLE]

Bounding the leading co-efficient in the failure exponent,

[TABLE]

.

Definition: Let us define a function $\epsilon(s)$ as follows:

[TABLE]

Note that $\epsilon(s)$ is an upper bound on the leading co-efficient of the failure exponent for an $(N_{1},N_{2},P,s)$ Coded Convolution. Now we compare coding with the uncoded strategy.

For the uncoded strategy in Section III, the failure event occurs exactly (not upper bound) when at most $P-1$ processors have failed to finish. The failure exponent can thus be computed exactly using steps similar to (13). For the uncoded strategy, the length of the vectors in each processor is $s=\sqrt{N_{1}N_{2}/P}$ and the number of processors to wait for is $P$ (also given by $K=P-\frac{Ps}{N_{2}}+\frac{N_{1}}{s}$ from Theorem 2). The leading co-efficient in the failure exponent is exactly given by,

[TABLE]

Theorem 4

For $N_{1}=\Theta(N_{2})$ and fixed Weibull parameter $\alpha$, there exists an $(N_{1},N_{2},P,s)$ Coded Convolution strategy with $s$ strictly greater than $\sqrt{N_{1}N_{2}/P}$ such that the ratio of the exponents of the asymptotic failure probability of the coded strategy to the naive uncoded strategy scales as $\Omega(\sqrt{P})$ and thus diverges to infinity for large $P$.

Proof:

From (14), the leading co-efficient in the failure exponent for the uncoded strategy is \epsilon\big{(}\sqrt{N_{1}N_{2}/P}\big{)}. Now let us choose a coded strategy with $s=2\sqrt{N_{1}N_{2}/P}$. Putting this value of $s$ in $\epsilon(s)$, we get $\epsilon(s)=-\frac{\mu^{\alpha}(\frac{3}{2}\sqrt{\frac{PN_{1}}{N_{2}}}+1)}{\left(4C\sqrt{\frac{N_{1}N_{2}}{P}}\log{4\sqrt{\frac{N_{1}N_{2}}{P}}}\right)^{\alpha}}$. Thus comparing the ratio,

[TABLE]

Since from the conditions of the theorem, $N_{1}=\Theta(N_{2})$, the ratio scales as $\Omega(\sqrt{P})$ and diverges for large $P$. ∎

Comment: The best choice of $s$ would be an integer in the range \big{(}\sqrt{N_{1}N_{2}/P},\min\{N_{1},N_{2}\}\big{]} that maximizes $\left|\epsilon(s)\right|$.

Comparison with Replication:

We show in Appendix C that the exponent of the probability of failure to meet a deadline in the asymptotic regime depends on the worst case $K$ for replication. In this paper, we include simulations in Fig. 4 showing that coded convolution outperforms replication in terms of the exponent of the probability of failure to meet a deadline. Moreover the exponents observed from the simulations are quite close to those calculated theoretically using the worst case $K$.

Simulation Results: We consider the convolution of a given vector of length $N_{1}=2^{12}$ with an unknown vector of length $N_{2}=2^{11}$, using $P=8$ parallel processors, based on an exponential model in MATLAB. Our results in Fig. 4 show that coded convolution has the fastest decay of failure exponent for large deadlines. The slopes obtained from simulations are found to be quite close to the theoretically calculated values.

VII Discussion

Thus, the proposed strategy outperforms existing strategies in terms of the exponent of the probability of failure to meet a deadline, in the limit of large deadlines. It might be observed that our strategy allows for online encoding and reconstruction as compared to existing works on coded matrix-vector products where the encoding cost can be amortized only if the matrix is pre-specified.

Acknowledgment

This work was supported in part by Systems on Nanoscale Information fabriCs (SONIC), one of the six SRC STARnet Centers, sponsored by MARCO and DARPA. The support of NSF Awards 1350314, 1464336 and 1553248 is also acknowledged. Sanghamitra Dutta also received Prabhu and Poonam Goel Graduate Fellowship.

The authors would like to thank Franz Franchetti, Tze Meng Low and Doru Thom Popovici for their useful feedback regarding this research. Praveen Venkatesh, Haewon Jeong and Yaoqing Yang are also thanked for their comments and suggestions.

Appendix A Optimal Uncoded Strategy

We prove that performing convolution under Scenario 2, results in equal or higher per processor complexity. Without loss of generality, assume that $s_{2}\leq s_{1}$. Then, under Scenario 2, the complexity of the convolution is given by $C(2s_{1})(\log{(2s_{2})}+1)$. Now, we are to solve

[TABLE]

Examine the function

[TABLE]

This function is concave in $s_{1}$. Let us examine its derivative.

[TABLE]

The derivative is positive in the range of $s_{1}$, i.e. $1\leq s_{1}\leq\min\{N_{1},\frac{N_{1}N_{2}}{P}\}$. Thus, the minimum is attained for the least value of $s_{1}$. However, as the product of $s_{1}$ and $s_{2}$ is constant, $s_{1}$ has to be greater or equal to the geometric mean, since otherwise $s_{2}>s_{1}$ which violates our assumption. Thus, the minimum value of $s_{1}$ is $\sqrt{\frac{N_{1}N_{2}}{P}}$, which results in the same time complexity as convolution without overlap methods.

Now, we consider performing convolutions using several serial uses of the given $P$ parallel processors, say $r$ uses. Then, effectively we have $rP$ processors, and the length of the convolution vector in each serial use of a single processor is $\sqrt{\frac{N_{1}N_{2}}{rP}}$. As this operation is to be done $r$ times, the computational complexity may be written as $r(2\sqrt{\frac{N_{1}N_{2}}{rP}})\log{(2\sqrt{\frac{N_{1}N_{2}}{rP}})}$, which increases roughly with a factor of $\sqrt{r}$, compared to convolution in a single shot. Thus, use of the processors serially, only worsens the computational complexity per processor, while still requiring all of them to finish.

Appendix B Encoding and Decoding Complexity

Theorem 3: For an $(N_{1},N_{2},P,s)$ Coded Convolution using a Vandermonde Encoding Matrix, the ratio of the total encoding complexity and reconstruction complexity to the per processor complexity tends to [math], as $N_{1},N_{2},P\to\infty$, when P(\log{P})^{2}=o\Big{(}\log{\sqrt{\frac{N_{1}N_{2}}{P}}}\Big{)}.

Proof:

Encoding For encoding, the first vector $\bm{a}$ is split into smaller pieces of length $s$, as given by $\{\bm{a}_{1},\bm{a}_{2},\dots,\bm{a}_{\frac{N_{1}}{s}}\}$. Now we encode the $\bm{a}_{i}$s as follows:

[TABLE]

Here $\bm{G}_{\frac{N_{1}}{s},\frac{Ps}{N_{2}}}$ is chosen as a Vandermonde matrix and $\{g_{1},g_{2},\dots,g_{\frac{Ps}{N_{2}}}\}$ are all distinct reals. Observe that the encoding process thus becomes the evaluation of polynomials of degree $\frac{N_{1}}{s}-1$( Note that $\frac{N_{1}}{s}<\frac{Ps}{N_{2}}$) at $\frac{Ps}{N_{2}}$ points, repeated $s$ times for $s$ length vector $\bm{a}_{i}$s. From [18], [19], it is known that the evaluation of a polynomial of degree $\frac{Ps}{N_{2}}-1$ at $\frac{Ps}{N_{2}}$ arbitrary points can be performed in $\mathcal{O}(\frac{Ps}{N_{2}}(\log{\frac{Ps}{N_{2}}})^{2})$. As this process is repeated $s$ times, the total encoding complexity is given by $\mathcal{O}\left(s\left(\frac{Ps}{N_{2}}\right)(\log{\frac{Ps}{N_{2}}})^{2}\right)$. Now observe that,

[TABLE]

Note that, (21) holds from the choice of $\sqrt{\frac{N_{1}N_{2}}{P}}\leq s$ which implies $\frac{Ps}{N_{2}}\leq P$. Thus, the encoding complexity is $\mathcal{O}(sP(\log{P})^{2})$.

Reconstruction The reconstruction complexity consists of the complexity of decoding and subsequent additions. First we compute the complexity of decoding.

For decoding, convolution outputs of length $2s$ arrive from each parallel processor. For reconstructing each of the $\frac{N_{2}}{s}$ parts of vector $\bm{x}$, any $\frac{N_{1}}{s}$ convolution outputs are sufficient. Recall that for any $\bm{x}_{j}$,

[TABLE]

If $\bm{G}$ is chosen as a Vandermonde Matrix, the reconstruction problem takes the form,

[TABLE]

The reconstruction of $\{\bm{a}_{i}\ast\bm{x}_{j}\}$ for $i=1,2,\dots,\frac{N_{1}}{s}$ thus reduces to the interpolation of a polynomial of degree $\frac{N_{1}}{s}-1$ from its value at $\frac{N_{1}}{s}$ known, arbitrary points, to be repeated $2s$ times, which is the length of the convolution output. From [18], [19], it is known that the interpolation of a polynomial of degree $\frac{N_{1}}{s}-1$ from its value at $\frac{N_{1}}{s}$ arbitrary points can be performed in $\mathcal{O}(\frac{N_{1}}{s}(\log{\frac{N_{1}}{s}})^{2})$. As this step is repeated $2s$ times for $\frac{N_{2}}{s}$ parts of vector $\bm{x}$, the total decoding complexity is given by $\mathcal{O}\left((2s)\left(\frac{N_{1}N_{2}}{s^{2}}\right)(\log{\frac{N_{1}}{s}})^{2}\right)$. Now observe that,

[TABLE]

Note that, (24) holds from the choice of $\sqrt{\frac{N_{1}N_{2}}{P}}\leq s$ which also implies $\frac{N_{1}}{s}\leq\frac{N_{1}N_{2}}{s^{2}}\leq P$.

Now, let us consider the complexity of the subsequent additions. Note that, we have a total of $\frac{N_{1}}{s}\frac{N_{2}}{s}$ vectors to be added with appropriate shifts. As each vector is of length $2s$, the complexity of adding any one vector is $\mathcal{O}(2s)$. Thus, the total complexity of the subsequent additions is given by \mathcal{O}\big{(}(2s)\frac{N_{1}}{s}\frac{N_{2}}{s}\big{)}. Now observe that,

[TABLE]

Note that, (25) also holds from the choice of $\sqrt{\frac{N_{1}N_{2}}{P}}\leq s$.

Thus, from (21), (24) and (25), we can bound the total encoding and reconstruction complexity as follows:

[TABLE]

Here $D$ is a constant. And (27) follows from the conditions of the theorem that $P(\log{P})^{2}=o\left(2s\log{\sqrt{\frac{N_{1}N_{2}}{P}}}\right)$ and from the choice of $\sqrt{\frac{N_{1}N_{2}}{P}}\leq s$. Thus the total reconstruction complexity, i.e., the complexity of decoding and subsequent additions is $o\left(2s\log{(2s)}\right)$. Recall that the per processor complexity for performing convolutions of two vectors of length $s$ is given by $C(2s)\log{(2s)}$. Thus, the ratio of the decoding complexity to the per-processor time complexity tends to [math] as $N_{1},N_{2},P\to\infty$. ∎

Appendix C Exponent for Repetition

In Section VI, we perform an asymptotic analysis of the exponent of the probability of failure to meet a deadline in the limit of the deadline diverging to infinity. Recall from Section VI that an upper bound for the leading coefficient of the failure exponent for Coded Convolution is given by

[TABLE]

Here $K$ is the number of processors one needs to wait for in the worst case for Coded Convolution. We also showed that for uncoded strategy, this upper bound is actually exactly equal to the leading co-efficient of the failure exponent (putting worst case $K=P$) as the deadline diverges to infinity. Now, we show that even for repetition strategy this upper bound is actually exactly equal to the upper bound using worst case $K$ as the deadline diverges to infinity.

Theorem 4: For a $(P,r)$ repetition strategy, the leading coefficient of the exponent in the probability of failure to meet a deadline converges as

[TABLE]

where $K=P-r+1$ is the number of processors to wait for in the worst case for a $(P,r)$ repetition strategy.

Proof:

Recall that in a $(P,r)$ repetition strategy, we basically divide the operation of convolution into $P/r$ equal tasks, and each task has $r$ repetitions, so as to use all the given $P$ processors. Let us introduce the notation $T_{i,j}$ to represent the random variable corresponding to the different computation times on different processors. Here $i$ is the task index which varies from $1$ to $P/r$ for the $P/r$ different parts of the convolution task. Also let $j$ be the replica index for each task which varies from $1$ to $r$ for $r$ replicas of each task. Thus $T_{i,j}$ represents the computation time of the $j$-th copy of the $i$-th task.

Consider any task with index $i$. Let $T^{\prime}_{i}$ denote the completion time of the particular task. The task finishes when any one of its $r$ replicas finish. Thus, the distribution of $T^{\prime}_{i}$ is given by

[TABLE]

Now let us try to find the distribution of $T^{\prime}_{i}$. Recall that each $T_{i,j}$ is a random variable denoting the time required to perform a convolution of length $s$ at a processor where $s=\sqrt{\frac{N_{1}N_{2}r}{P}}$. Its computational complexity is given by $2Cs\log(2s)$. From shifted Weibull model assumptions, the c.d.f. of each $T_{i,j}$ is given by

[TABLE]

Since the $\{T_{i,j}\}$s are i.i.d., we thus have

[TABLE]

Let us denote $\tilde{F}_{s}(t)=\Pr(T^{\prime}_{i}<t)$. Now, for repetition the probability of failure to meet a deadline occurs when at least one of the $P/r$ tasks have not completed. Thus the failure probability can be exactly computed as:

[TABLE]

Here $c(P,K)$ denotes a function of $P$ and $K(=P-r+1)$, that is independent of $t$ (Note that $K=P-r+1$ for repetition). The last line follows since for $t$ large enough, $\frac{(\tilde{F}_{s}(t))}{(1-\tilde{F}_{s}(t))}\gg 1$, and thus for the purpose of analysis of the failure exponent for large $t$, the binomial summation is of the same order as the largest term, as we show here:

[TABLE]

This implies,

[TABLE]

Here $c_{1}(P,K)=\binom{P/r}{P/r-1}$, i.e., a function of $P$ and $K(=P-r+1)$ only and independent of $t$. Similarly, $c_{2}(P,K)=\sum_{l=0}^{\frac{P}{r}-1}\binom{P/r}{l}$ is also a function of $P$ and $K(=P-r+1)$ only and independent of $t$. Thus,

[TABLE]

Bounding the leading co-efficient in the failure exponent,

[TABLE]

This proves the theorem. ∎

Appendix D Heuristic analysis of exponents

Statement: There exists an $(N_{1},N_{2},P,s)$ Coded Convolution strategy that outperforms the naive uncoded strategy in terms of asymptotic probability of failure if $P(\log{P})^{2}=o\left(\log{\sqrt{\frac{N_{1}N_{2}}{P}}}\right)$ and $\alpha<2\sqrt{\frac{PN_{1}}{N_{2}}}\left(1+\frac{1}{\log{2\sqrt{\frac{N_{1}N_{2}}{P}}}}\right)^{-1}$.

Justification: Consider the function given by

[TABLE]

A higher value of $E(s)$ implies faster decay of the failure exponent. Note that $s$ only takes values in the range $\sqrt{\frac{N_{1}N_{2}}{P}}\leq s\leq\min\{N_{1},N_{2}\}$. Here $s=\sqrt{\frac{N_{1}N_{2}}{P}}$ corresponds to the uncoded strategy, and any $s$ strictly greater than $\sqrt{\frac{N_{1}N_{2}}{P}}$ lies in the coded regime.

Now, we compare the exponents for uncoded and coded strategies.

Consider the derivative of $E(s)$ in this range, as given by,

[TABLE]

At $s=\sqrt{\frac{N_{1}N_{2}}{P}}$ (Uncoded), the derivative is given by,

[TABLE]

If $E^{\prime}(s)|_{\sqrt{\frac{N_{1}N_{2}}{P}}}>0$, then $E(s)$ is strictly increasing, as $s$ increases beyond $\sqrt{\frac{N_{1}N_{2}}{P}}$, i.e., into the coded regime. Thus, $E(s)$ attains a maximum in the regime $s>\sqrt{\frac{N_{1}N_{2}}{P}}$, i.e., in the coded regime. Thus, it is sufficient that

[TABLE]

This implies,

[TABLE]

Thus there exists an $s$ in the coded regime for which the asymptotic failure probability is lesser than that in the uncoded regime.

Note that, $\alpha=1$, i.e., the shifted exponential also belongs to this regime. We also show some plots for the special case of $N_{1}=4N$, $N_{2}=N$ and $P=\sqrt{\log{N}}$, showing that choosing an $s$ in the coded regime outperforms uncoded strategy for the chosen values of $\alpha$ in Fig. 5.