CodedSketch: A Coding Scheme for Distributed Computation of Approximated Matrix Multiplication

TL;DR

CodedSketch introduces a novel distributed matrix multiplication scheme that combines count-sketch compression with structured coding to efficiently approximate results while reducing the number of worker nodes needed for recovery.

Contribution

The paper presents a new coding scheme that leverages count-sketch and structured codes to minimize recovery threshold in distributed approximate matrix multiplication.

Findings

Reduces recovery threshold compared to existing methods.

Provides theoretical analysis of approximation accuracy.

Achieves efficient distributed computation with controlled error.

Abstract

In this paper, we propose CodedSketch, as a distributed straggler-resistant scheme to compute an approximation of the multiplication of two massive matrices. The objective is to reduce the recovery threshold, defined as the total number of worker nodes that we need to wait for to be able to recover the final result. To exploit the fact that only an approximated result is required, in reducing the recovery threshold, some sorts of pre-compression are required. However, compression inherently involves some randomness that would lose the structure of the matrices. On the other hand, considering the structure of the matrices is crucial to reduce the recovery threshold. In CodedSketch, we use count--sketch, as a hash-based compression scheme, on the rows of the first and columns of the second matrix, and a structured polynomial code on the columns of the first and rows of the second matrix. This arrangement allows us to exploit the gain of both in reducing the recovery threshold. To increase the accuracy of computation, multiple independent count--sketches are needed. This independency allows us to theoretically characterize the accuracy of the result and establish the recovery threshold achieved by the proposed scheme. To guarantee the independency of resulting count--sketches in the output, while keeping its cost on the recovery threshold minimum, we use another layer of structured codes.

Tables

Table 1. TABLE I: The Count-Sketch results of 𝐂 𝐂 \mathbf{C}

Index	0	1	2
1st Count-Sketch			${𝐏}_{23}^{(1,1)}=-{𝐂}_{2,0}-{𝐂}_{2,2}$
2nd Count-Sketch
3rd Count-Sketch	$\begin{array}{c}{𝐏}_{03}^{(3,3)}=-{𝐂}_{2,1}-{𝐂}_{2,2}+{𝐂}_{2,3}\end{array}$

Equations

$$\mathbf{P}(\bigcap _{i=1}^{|\mathcal{S}|}{X}_i=x_i)=\prod _{i=1}^{|\mathcal{S}|}\mathbf{P}(X_i=x_i).$$

$$\mathbf{P}(h(x_1)=h(x_2)=...=h(x_k))\le \frac{1}{b^{k-1}}.$$

$$\mathbf{P}((h(x_1)=y_1)\cap (h(x_2)=y_2)\cap ...\cap (h(x_k)=y_k))=\frac{1}{b^k}.$$

$$c_{h(i)}=c_{h(i)}+s(i)a_i,$$

$$\mathbf{E}[c_{t,i}]$$

$$\mathbf{Var}[c_{t,i}]$$

$$\mathbf{P}\bigg{[}|c_{t,i}-a_{i}|\geq\epsilon{\left\|\mathbf{a}\right\|_{2}}\bigg{]}\leq\frac{1}{3}$$

$$\mathbf{P}\bigg{[}|\textnormal{median}\{c_{1,i},c_{2,i},...,c_{d,i}\}-a_{i}|\geq\epsilon{{{\left\|\mathbf{a}\right\|}_{2}}}\bigg{]}\leq\delta.$$

$$\mathbf{P}\bigg{[}|\tilde{a}_{i}-{a}_{i}|\geq\frac{\epsilon}{\sqrt{\kappa}}\mathrm{err}_{2}^{(\kappa)}(\mathbf{a})\bigg{]}\leq 2e^{-\Omega(d\epsilon^{2})},$$

$$p_{\mathbf{a}}(\alpha )=\sum _{i=0}^{n-1}s(i)a_i{\alpha }^{h(i)},$$

$$p_{\mathbf{a}}(\alpha )$$

$$p_{\mathbf{b}}(\alpha )$$

$$p_{\mathbf{a}{\mathbf{b}}^T}(\alpha )=\sum _{i,j}s(i,j){[\mathbf{a}{\mathbf{b}}^T]}_{ij}{\alpha }^{h(i,j)}.$$

$$s(i,j)=s_1(i)s_2(j).$$

$$h(i,j)=h_1(i)+h_2(j).$$

$$\mathbf{E}[\tilde{\mathbf{C}}]=\mathbf{C}.$$

$$\displaystyle\mathbf{P}\bigg{[}|[\mathbf{C}]_{i,j}-[\tilde{\mathbf{C}}]_{i,j}|\geq{\epsilon}||\mathbf{C}||_{F,(i^{\prime},j^{\prime})}\bigg{]}\leq\delta\hskip 28.45274pt\forall i\in[0:r-1],\forall j\in[0:t-1],$$

$$||\mathbf{C}|{|}_{F,(i^{\prime },j^{\prime })}\triangleq (\sum _{k=0}^{m-1}\sum _{\ell =0}^{n-1}[{\mathbf{C}}_{k,\ell }{]}_{i^{\prime },j^{\prime }}^2{)}^{\frac{1}{2}}.$$

$$\mathbf{A}={\left[{\begin{array}[]{*{20}{c}}{{\mathbf{A}_{0,0}}}&{{\mathbf{A}_{0,1}}}&{...}&{{\mathbf{A}_{0,p-1}}}\\ {{\mathbf{A}_{1,0}}}&{{\mathbf{A}_{1,1}}}&{...}&{{\mathbf{A}_{1,p-1}}}\\ \vdots&\vdots&\ddots&\vdots\\ {{\mathbf{A}_{m-1,0}}}&{{\mathbf{A}_{m-1,1}}}&\cdots&{{\mathbf{A}_{m-1,p-1}}}\end{array}}\right]_{r\times s}},$$

$$\mathbf{B}={\left[{\begin{array}[]{*{20}{c}}{{\mathbf{B}_{0,0}}}&{{\mathbf{B}_{0,1}}}&{...}&{{\mathbf{B}_{0,n-1}}}\\ {{\mathbf{B}_{1,0}}}&{{\mathbf{B}_{1,1}}}&{...}&{{\mathbf{B}_{1,n-1}}}\\ \vdots&\vdots&\ddots&\vdots\\ {{\mathbf{B}_{p-1,0}}}&{{\mathbf{B}_{p-1,1}}}&\cdots&{{\mathbf{B}_{p-1,n-1}}}\end{array}}\right]_{s\times t}},$$

$${\tilde{\mathbf{A}}}_i={\mathcal{E}}_i(\mathbf{A}),$$

$${\tilde{\mathbf{B}}}_i={\tilde{\mathcal{E}}}_i(\mathbf{B}),$$

$$\tilde{\mathbf{C}}=\mathcal{D}(\{{\tilde{\mathbf{C}}}_i{\}}_{i\in \mathcal{K}}).$$

$$N^{\ast }(ϵ,\delta ,p,m,n)\le \min \left\{(2p\lceil \frac{3}{{ϵ}^2}\rceil -1)(2\lceil \log \frac{1}{\delta }\rceil -1)-1,pmn+p-1\right\}.$$

$$\min \left\{(2p\lceil \frac{3\kappa }{{ϵ}^2}\rceil -1)(2\lceil \log mn\rceil -1)-1,pmn+p-1\right\}.$$

$$\mathbf{A}=\left[{\begin{array}[]{*{20}{c}}{{\mathbf{A}_{0,0}}}&{{\mathbf{A}_{0,1}}}&{{\mathbf{A}_{0,2}}}&{{\mathbf{A}_{0,3}}}\\ {{\mathbf{A}_{1,0}}}&{{\mathbf{A}_{1,1}}}&{{\mathbf{A}_{1,2}}}&{{\mathbf{A}_{1,3}}}\\ {{\mathbf{A}_{2,0}}}&{{\mathbf{A}_{2,1}}}&{{\mathbf{A}_{2,2}}}&{{\mathbf{A}_{2,3}}}\\ {{\mathbf{A}_{3,0}}}&{{\mathbf{A}_{3,1}}}&{{\mathbf{A}_{3,2}}}&{{\mathbf{A}_{3,3}}}\end{array}}\right]_{r\times s},$$

$$\mathbf{B}=\left[{\begin{array}[]{*{20}{c}}{{\mathbf{B}_{0,0}}}&{{\mathbf{B}_{0,1}}}&{{\mathbf{B}_{0,2}}}&{{\mathbf{B}_{0,3}}}\\ {{\mathbf{B}_{1,0}}}&{{\mathbf{B}_{1,1}}}&{{\mathbf{B}_{1,2}}}&{{\mathbf{B}_{1,3}}}\\ {{\mathbf{B}_{2,0}}}&{{\mathbf{B}_{2,1}}}&{{\mathbf{B}_{2,2}}}&{{\mathbf{B}_{2,3}}}\\ {{\mathbf{B}_{3,0}}}&{{\mathbf{B}_{3,1}}}&{{\mathbf{B}_{3,2}}}&{{\mathbf{B}_{3,3}}}\end{array}}\right]_{s\times t}.$$

$$\displaystyle\hat{\mathbf{A}}(x)=\left[{\begin{array}[]{*{20}{c}}{{{\hat{\mathbf{A}}}_{0,0}}(x)}\\ {{{\hat{\mathbf{A}}}_{1,0}}(x)}\\ {{{\hat{\mathbf{A}}}_{2,0}}(x)}\\ {{{\hat{\mathbf{A}}}_{3,0}}(x)}\end{array}}\right]$$

$$\displaystyle=\left[{\begin{array}[]{*{20}{c}}{{\mathbf{A}_{0,0}}+x{\mathbf{A}_{0,1}}+{x^{2}}{\mathbf{A}_{0,2}}+{x^{3}}{\mathbf{A}_{0,3}}}\\ {{\mathbf{A}_{1,0}}+x{\mathbf{A}_{1,1}}+{x^{2}}{\mathbf{A}_{1,2}}+{x^{3}}{\mathbf{A}_{1,3}}}\\ {{\mathbf{A}_{2,0}}+x{\mathbf{A}_{2,1}}+{x^{2}}{\mathbf{A}_{2,2}}+{x^{3}}{\mathbf{A}_{2,3}}}\\ {{\mathbf{A}_{3,0}}+x{\mathbf{A}_{3,1}}+{x^{2}}{\mathbf{A}_{3,2}}+{x^{3}}{\mathbf{A}_{3,3}}}\end{array}}\right],$$

$$\displaystyle\hat{\mathbf{B}}(x)^{T}={\left[{\begin{array}[]{*{20}{c}}{\hat{\mathbf{B}}_{0,0}(x)}\\ {\hat{\mathbf{B}}_{0,1}(x)}\\ {\hat{\mathbf{B}}_{0,2}(x)}\\ {\hat{\mathbf{B}}_{0,3}(x)}\end{array}}\right]^{T}}$$

$$\displaystyle={\left[{\begin{array}[]{*{20}{c}}{{x^{3}}\mathbf{B}_{0,0}^{T}+{x^{2}}\mathbf{B}_{1,0}^{T}+x\mathbf{B}_{2,0}^{T}+\mathbf{B}_{3,0}^{T}}\\ {{x^{3}}\mathbf{B}_{0,1}^{T}+{x^{2}}\mathbf{B}_{1,1}^{T}+x\mathbf{B}_{2,1}^{T}+\mathbf{B}_{3,1}^{T}}\\ {{x^{3}}\mathbf{B}_{0,2}^{T}+{x^{2}}\mathbf{B}_{1,2}^{T}+x\mathbf{B}_{2,2}^{T}+\mathbf{B}_{3,2}^{T}}\\ {{x^{3}}\mathbf{B}_{0,3}^{T}+{x^{2}}\mathbf{B}_{1,3}^{T}+x\mathbf{B}_{2,3}^{T}+\mathbf{B}_{3,3}^{T}}\end{array}}\right]^{T}}.$$

$${\hat{\mathbf{C}}}_{k,\ell }(x)={\hat{\mathbf{C}}}_0^{(k,\ell )}+{\hat{\mathbf{C}}}_1^{(k,\ell )}x+...+{\hat{\mathbf{C}}}_6^{(k,\ell )}{x}^6.$$

$$\displaystyle{\mathbf{F}_{1}}(x,\alpha)=\big{(}-{\hat{\mathbf{A}}_{0,0}}(x)+{\hat{\mathbf{A}}_{1,0}}(x)+{\hat{\mathbf{A}}_{3,0}}(x)\big{)}-{\hat{\mathbf{A}}_{2,0}}(x)\alpha,$$

Full text

I Introduction

Linear operations, often represented by matrix multiplication, are the key techniques used in many applications such as optimization and machine learning. Many such applications require processing large-scale matrices. For example, in the deep neural networks, the convolution layers, which are operations based on matrix multiplication, account for a large fraction of computational time [2]. To increase the accuracy of the learning, we increase the size of the model, by using more layers and more neurons in each layer. This would increase the computation complexity and the overall training time. This heavy computation cannot be completed over a single machine. An inevitable solution to overcome this challenge is to distribute computation over several machines[3, 4]. In particular, consider a master-worker setting for distributed computing, in which the master node has access to the information of two matrices. The master then partitions the task of matrix multiplications into some smaller sub-tasks and assigns each sub-task to one of the worker nodes to be executed. In this setting, the execution time is dominated by the speed of the slowest worker nodes, or the stragglers. This is one of the main challenges in distributed computing [5].

Conventionally, the effect of stragglers has been mitigated by using redundancy in computing. This means that every task is executed over more than one machine, and the results are fetched from the fastest ones. More recently, it is shown that coding can be more effective in coping with stragglers in terms of minimizing the recovery threshold, defined as the total number of worker nodes that the master node needs to wait for to be able to recover the final result [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. In [6, 7], it is suggested that in matrix multiplication, one or both of the matrices are coded separately using maximum distance separable (MDS) codes. In [8], polynomial codes have been proposed to code each matrix, such that the result of the multiplications across worker nodes become also MDS coded. The short-dot technique has been presented in [11], where coded redundancy is added to the computation, and the effort is to keep the coded matrices sparse and thus reduce the load of the computation. In [9], an extension of the polynomial codes, known as entangled polynomial codes, has been proposed that admits flexible partitioning of each matrix and minimizes the number of unwanted computations. The general version of entangled polynomial codes is proposed in [10], this means a coding scheme is introduced to multiply more than two matrices. In [15], coded sparse matrix multiplication is proposed for the case where the result of the multiplication is sparse. In [12, 13], coded schemes have been used to develop multi-party computation scheme to calculate arbitrary polynomials of massive matrices, while preserving privacy of the data. In [16], a universal coding technique, Lagrange Code, is developed to code across several parallel computations of an arbitrary polynomial function, without communication across worker nodes. In [17], a coded computing framework is proposed to mitigate the effect of stragglers in the distributed gradient descent algorithm. Since in SGD, the algorithm works even with the approximate result, in [18] an approximate variant of the gradient coding in a straggler-resistant distributed setting is introduced using expander graphs. In [19], a fundamental trade-off among the computation load, error of computed gradient, and the number of stragglers is proposed. Similarly, there are many applications which process large-scale matrices, and computing the exact result of matrix multiplication is not required. In those cases, computing approximated matrix multiplication may significantly decrease the required number of worker nodes and computation load. To derive an approximated result of the matrix multiplication over a single server, linear sketching as a randomized method can be applied [20, 21, 22]. Linear sketching is an approach to reduce the dimension of the input matrices by multiplying them by a random matrix with certain properties. Also, in [23], it is proposed to use the count-sketch method, concatenated with fast Fourier transform to reduce multiplication complexity. Count-Sketch is a hashing-based approach of developing linear sketching to reduce the size of the input vector such that it can be recovered with a certain accuracy. Count-sketch method can be particularly efficient in massive computing, machine learning and statistics [24, 25]. To review count-sketch method, see [26, 27], or Subsection II-A on preliminaries in this paper. In [14], the OverSketch method has been proposed where some extra count-sketches are used, as a redundancy, to mitigate stragglers in a distributed setting (see Remark 10 for further discussion).

In this paper, we propose a distributed straggler–resistant computation scheme to achieve an approximation of the multiplication of two large matrices $\mathbf{A}\in\mathbb{R}^{r\times s}$ and $\mathbf{B}\in\mathbb{R}^{s\times t}$, where $r,s,t\in\mathbb{N}$ (see Fig. 1). The motivation is that in some optimization and machine learning algorithms which are based on stochastic gradient descent such as the learning phase of deep learning, the scheme is less sensitive to the accuracy of the result as long as it is an unbiased estimation of it. To exploit the fact that an approximated result is required to reduce the recovery threshold, we need to use some sorts of pre-compression. However, compression inherently involves some randomness that would lose the structure of the matrices. On the other hand, considering the structure of the matrices is crucial to reduce the recovery threshold. In this paper, we use the ideas of entangled polynomial codes and count-sketch to form a right combination of randomness and structured codes. To be specific, we use entangled polynomial codes on the columns of $\mathbf{A}$ and the rows of $\mathbf{B}$. Then, we use count–sketch compression on the rows of $\mathbf{A}$ and the columns of $\mathbf{B}$. This arrangement allows us to enjoy the benefits of both in reducing the recovery threshold. To improve the overall accuracy, we need to use multiple count–sketches. Another layer of structured codes, i.e., improved entangled polynomial codes, allows us to keep multiple count–sketches independent. This independency is used to prove theoretical guarantee on the performance of the final result and establish an achievable recovery threshold.

Notation: For $n_{1},n_{2}\in\mathbb{Z}$, the notation $[n_{1}:n_{2}]$ represents the set $\{n_{1},n_{1}+1,...,n_{2}\}$. The cardinality of a set $\mathcal{S}$ is denoted by $|\mathcal{S}|$. In addition, $\mathbf{E}[X]$ refers to the expected value of $X$. The $i$th element of a vector $\mathbf{v}$, is denoted by ${v}_{i}$ and the $(i,j)$-th entry of a matrix $\mathbf{A}$ is denoted by $[\mathbf{A}]_{i,j}$. Also, $\mathbf{A}_{k,\ell}$ is block $(k,\ell)$ of block matrix $\mathbf{A}$, and $[\mathbf{A}_{k,\ell}]_{i,j}$ is entry $(i,j)$ in the block $(k,\ell)$ of block matrix $\mathbf{A}$. $||\mathbf{A}||_{F}$ is the Frobenius norm of matrix $\mathbf{A}$.

II Problem Formulation, Notation and Preliminaries

II-A Preliminaries

In this subsection, we review some preliminaries.

Definition 1** ($k$-wise Independent Random Variables [28]).**

A set of discrete random variables $X_{1},X_{2},...,X_{n}$ is $k$-wise independent if for any subset $\mathcal{S}$ of these random variables, $|\mathcal{S}|\leq k$, and for any values $x_{i}$, $i\in[1:|\mathcal{S}|]$ we have

[TABLE]

Definition 2** (Hash Function [28]).**

A hash function is a function that maps a universe $\mathcal{U}$ into a special range $[0:b-1]$ for some $b\in\mathbb{N}$, i.e., $h:\mathcal{U}\to[0:b-1]$. In other words, a hash function operates as a method in which items from the universe are placed into $b$ bins.

The most simple family of hash functions is completely random hash functions which can be analyzed using a model known as balls and bins model. The hash values $h(x_{1}),h(x_{2}),...,h(x_{n})$ are considered independent and uniform over the range of the hash function for any collection of data $x_{1},x_{2},...,x_{n}$. Since this class of hash functions is expensive in terms of computing and storage cost, it is not very useful in practice. Another family of hash functions is known as Universal family, which satisfies some provable performance guarantees [28].

Definition 3** ($k$-universal Hash Family [28]).**

The hash family $\mathcal{H}$ is defined as a $k$-universal family if for any hash function $h:\mathcal{U}\to[0:b-1]$ with $|\mathcal{U}|\geq b$, chosen uniformly at random from $\mathcal{H}$, and for any collection of $x_{1},x_{2},...,x_{k}$, we have

[TABLE]

In addition $\mathcal{H}$ is defined as a strongly $k$-universal family if for any values $y_{1},...,y_{k}\in[0:b-1]$ we have

[TABLE]

This definition implies that for any fix $x\in\mathcal{U}$, $h(x)$ is uniformly distributed over $[0:b-1]$. Also, if $h$ is chosen uniformly at random from $\mathcal{H}$, for any distinct $x_{1},x_{2},...,x_{k}$, the values of $h(x_{1}),h(x_{2}),...,h(x_{k})$ are independent.

Remark 1*:*

The values $h(x_{1}),h(x_{2}),...,h(x_{k})$ are $k$-wise independent if the hash function $h$ is chosen from a strongly $k$-universal family.

Definition 4** (Count-Sketch [26, 27]).**

The count-sketch is a method for representing a compact form of data which maps an $n$-dimensional vector $\mathbf{a}$ to a $b$-dimensional vector $\mathbf{c}$ where $n\gg b$. The count-sketch of $\mathbf{a}$ can be defined using an iterative process, initialized with the vector $\mathbf{c}$ to be entirely zero, as follows: in the $i$th iteration ($i=[0:n-1]$) we have

[TABLE]

where $s:[0:n-1]\to\{-1,1\}$ is a sign function, $a_{i}$ is $i$th entry of $\mathbf{a}$, and $h:[0:n-1]\to[0:b-1]$ is a hash function that $h(i)$ computes the hash of $i$.

One count-sketch of $\mathbf{a}$ is created by using one hash function and one sign function. To improve accuracy in this approximation, more than one hash function can be used (say $d\in\mathbb{N}$ hash functions). To create $d$ count-sketches, we need $d$ pairwise independent sign functions and $d$ 2-wise independent hash functions. The output of these $d$ count-sketches is a $d\times b$ matrix. $a_{i}$ as the entry $i$ of $\mathbf{a}$, for each $i\in[0:n-1]$, can be approximately recovered by taking the median of the values of some entries of this matrix. For details, see Algorithm 1.

Theorem 1** ([26, 29]).**

If $d\geq\log\frac{1}{\delta}$ and $b\geq\frac{3}{\epsilon^{2}}$ then for entry $a_{i}$ of the input vector $\mathbf{a}$, $i\in[0:n-1]$, we have

[TABLE]

and

[TABLE]

*where $c_{t,i}=s_{t}(i)[\mathbf{C}]_{t,h_{t}(i)}$ is an estimation of $a_{i}$. *

Corollary 2** ([26, 29]).**

If there are $d$ estimations like $c_{t,i}$, then

[TABLE]

Corollary 2 shows that by choosing $d=\mathcal{O}(\log n)$ with probability at least $1-\frac{1}{\text{poly}(n)}$, we have $|\tilde{a}_{i}-a_{i}|\leq\epsilon{{{\left\|\mathbf{a}\right\|}_{2}}}$, where $\tilde{a}_{i}=\text{median}\{c_{1,i},c_{2,i},...,c_{d,i}\}$, for $i\in[0:n-1]$.

Remark 2*:*

Suppose $n$-dimensional vector $\mathbf{a}$ has only $\kappa$ nonzero entries. Then, from Corollary 2, one can see that an approximated version of $\mathbf{a}$ with accuracy $\epsilon=\mathcal{O}(\frac{1}{\sqrt{\kappa}})$ can be achieved by $d=\mathcal{O}(\log n)$ count-sketches.

Theorem 3** ([30, 31, 29]).**

For an $n$-dimensional vector $\mathbf{a}$ and $\kappa\in[0:n]$, if we choose the length of the count-sketches as $b\geq\frac{3\kappa}{\epsilon^{2}}$ and the number of count-sketches as $d$ then we have

[TABLE]

where $\mathrm{err}_{2}^{(\kappa)}(\mathbf{a})={(\sum_{i\notin\mathcal{S}}{|a_{i}|^{2}})}^{\frac{1}{2}}$ and $\mathcal{S}$ is the set of indices of the $\kappa$ largest (in absolute value) entries of $\mathbf{a}$.

Remark 3*:*

According to Theorem 3, if the $n$-dimensional vector $\mathbf{a}$ has only $\kappa$ nonzero entries and $d=\mathcal{O}(\log n)$, then the output of the count-sketch method computes $\mathbf{a}$ exactly with probability $1-\frac{1}{n^{\Theta(1)}}$.

Definition 5**.**

The count-sketch of an $n$-dimensional vector $\mathbf{a}$ can be represented by a polynomial, named sketch polynomial, as

[TABLE]

where $h:[0:n-1]\to[0:b-1]$ is the 2-wise independent hash function and $s:[0:n-1]\to\{-1,1\}$ is the 2-wise independent sign function, used to develop the count-sketch.

Let us assume that we use $h_{1}$ and $s_{1}$ to count-sketch vector $\mathbf{a}$ and $h_{2}$ and $s_{2}$ to count-sketch vector $\mathbf{b}$, respectively, represented by sketch polynomials

[TABLE]

then ${p_{\mathbf{a}\mathbf{b}^{T}}}(\alpha)\triangleq{p_{\mathbf{a}}}(\alpha){p_{\mathbf{b}}}(\alpha)$ represents a sketch polynomial for matrix $\mathbf{a}\mathbf{b}^{T}$, as

[TABLE]

where

[TABLE]

Also, the hash function on pairs $(i,j)$ is

[TABLE]

Remark 4*:*

Recall that $h_{1}(i)$ and $h_{2}(j)$ are two independent random variables with some distributions. Thus the distribution of $h(i,j)$ over its output range is the convolution of the distributions of $h_{1}(i)$ and $h_{2}(j)$.

II-B Problem Formulation

Consider a distributed system including a master and $N$ worker nodes, where the master node is connected to each worker node. Assume that the master node wants to compute $\tilde{\mathbf{C}}$ which is an approximation of the multiplication of two matrices $\mathbf{C}=\mathbf{A}\mathbf{B}$, where $\mathbf{A}\in\mathbb{R}^{r\times s}$, $\mathbf{B}\in\mathbb{R}^{s\times t}$, for some positive integers $m,n$ and $p$, are partitioned into $m\times p$ and $p\times n$ sub-matrices of equal size, respectively. Our goal is to compute $\tilde{\mathbf{C}}$ subject to the following conditions

Unbiasedness:

[TABLE] 2. 2.

$(\epsilon,\delta)$-accuracy:

We say $\tilde{\mathbf{C}}$ is an $(\epsilon,\delta)$-accurate approximation of matrix $\mathbf{C}$ if

[TABLE]

where $i^{\prime}=i\mod\frac{r}{m}$, $j^{\prime}=j\mod\frac{t}{n}$, and $[\mathbf{C}]_{i,j}$ is the $(i,j)$th entry of $\mathbf{C}$. Also, we define

[TABLE]

Remark 5*:*

In (16) and (17), the randomness is over the count-sketches which are used in the proposed scheme.

Suppose $\mathbf{A}$ and $\mathbf{B}$ are partitioned into the sub-matrices of equal size as

[TABLE]

[TABLE]

where $m,n$ and $p$ are positive integers. One of the constraints in this system is the limited storage of each worker. We assume that the size of the storage at each node is equal to $\frac{1}{pm}$ fraction of $\mathbf{A}$ plus $\frac{1}{pn}$ fraction of $\mathbf{B}$. Assume $\mathcal{E}_{i}:\mathbb{R}^{r\times s}\to\mathbb{R}^{\frac{r}{m}\times\frac{s}{p}}$ and $\tilde{\mathcal{E}}_{i}:\mathbb{R}^{s\times t}\to\mathbb{R}^{\frac{s}{p}\times\frac{t}{n}}$ are the encoding functions that are used by the master node to compute $\tilde{\mathbf{A}}_{i}$ and $\tilde{\mathbf{B}}_{i}$ for $i$th worker node, $i=1,...,N$, respectively. In other words, the master node encodes the input matrices and sends two coded matrices to $i$th worker node as follows:

[TABLE]

where $\tilde{\mathbf{A}}_{i}\in\mathbb{R}^{\frac{r}{m}\times\frac{s}{p}}$ and $\tilde{\mathbf{B}}_{i}\in\mathbb{R}^{\frac{s}{p}\times\frac{t}{n}}$. Each worker node computes $\tilde{\mathbf{C}}_{i}\triangleq\tilde{\mathbf{A}}_{i}\tilde{\mathbf{B}}_{i}$ and sends back the result to the master node. The problem formulation so far limits the communication and computation load per node. After receiving the results from a subset $\mathcal{K}\subset[N]$ of worker nodes, the master node can recover $\tilde{\mathbf{C}}$, the approximation of the original result. Let $\mathcal{D}(.)$ be a reconstruction function which operates on a subset $\mathcal{K}$ of the workers’ results and calculates $\tilde{\mathbf{C}}$ as follows

[TABLE]

Definition 6**.**

The recovery threshold of a distributed system with $N$ worker nodes and one master node is the minimum number of workers that the master node needs to wait for such that it is guaranteed the master node can complete the computation, subject to space constraint at each worker node and unbiasedness and $(\epsilon,\delta)$-accuracy conditions. In this paper, the optimum recovery threshold is denoted by $N^{*}(\epsilon,\delta,p,m,n)$.

III Main results

The following theorems state the main results of the paper.

Theorem 4**.**

For $(\epsilon,\delta)$-accurate approximation of the multiplication of two matrices, we have

[TABLE]

The proof of Theorem 4 is detailed in Subsection IV-B.

Remark 6*:*

To prove Theorem 4, we propose a coding scheme that benefits from three different features:

It exploits the fact that only an approximation result is needed, by using some count-sketches to pre-compress the input matrices which reduces the recovery threshold. 2. 2.

It relies on the structure of the matrix multiplication to add coded redundancy in the computation and mitigate the effect of stragglers. On the other hand, this coding can force the system to execute some unwanted calculations. The coding is designed such that the total number of unwanted calculations is reduced while the coding is maximum distance separable (MDS). 3. 3.

The proposed scheme, in the end, creates some independent count-sketches of $\mathbf{A}\mathbf{B}$, from which we calculate the final result. As a side product, some dependent count-sketches are also created. To minimize the recovery threshold, we need to minimize the number of these side products. Another layer of structured code is used to reduce the number of these side products and thus reduce the recovery threshold.

We note that between opportunities one and two, the first one injects some randomness to the input matrices, while the second one relies on their structure. To achieve both at the same time, we use count-sketches on the rows of the first and columns of the second matrix, and use a structured code on the columns of the first and the rows of the second matrix.

The code structure is motivated by count-sketch [26, 27], polynomial codes [8], entangled polynomial codes [9], MatDot codes [10], and improved entangled polynomial codes.

Remark 7*:*

The proposed scheme achieves the first term in Theorem 4. The second term is achieved by the entangled polynomial code [9] to calculate the exact result.

Remark 8*:*

The proposed scheme computes the approximated matrix multiplication in a straggler-resistant distributed system using one-shot coded computation, this means in this scheme one encoding, and one decoding step is used which will be explained below. In this scheme, no communication between worker nodes is needed, and the master node only sends two coded matrices to each worker node, and each worker node returns its computation result to the master node without any additional communication between worker nodes.

Remark 9*:*

In the proposed scheme, the amount of computation needed at the master node to find the approximate multiplication result of two $n\times n$ matrices scales $\mathcal{O}(n^{2})$. Also, the problem formulation restricts the computational complexity of each worker node which means a fixed computation load is specified to each worker node, and it only receives $\frac{1}{pm}$ of the matrix $\mathbf{A}$ and $\frac{1}{pn}$ of the matrix $\mathbf{B}$ and returns the multiplication of these two matrices to the master node. The computational complexity of the proposed scheme in the master node and each worker node is detailed in Subsection IV-C.

Theorem 5**.**

If matrix $\mathbf{C}=\mathbf{A}\mathbf{B}$ is $\kappa$-sparse, i.e., for all $i\in[0:\frac{r}{m}-1]$ and $j\in[0:\frac{t}{n}-1]$, there exist at most $\kappa$ non-zero elements among the $(i,j)$th entries of all $mn$ blocks of $\mathbf{C}$, then the proposed method computes $\mathbf{C}$ exactly with probability $1-\delta=1-\frac{1}{(mn)^{\Theta(1)}}$ and with the recovery threshold of

[TABLE]

The proof of Theorem 5 is detailed in Subsection IV-B.

IV The Proposed Scheme

In the following, the goal is to compute the approximation of the matrix multiplication over a distributed system using CodedSketch scheme.

IV-A Motivating Example

We first demonstrate the main idea of our scheme through a simple example. We then generalize this approach in the next section. Consider a distributed system with one master node and $N$ worker nodes which aim to collaboratively compute $\tilde{\mathbf{C}}$ as an approximation of $\mathbf{C}=\mathbf{A}\mathbf{B}$ where $\mathbf{A}$ and $\mathbf{B}$ are two matrices partitioned as follows

[TABLE]

[TABLE]

The result of the multiplication can be computed using summation of four outer products as $\mathbf{C}=\sum_{k=0}^{3}{\mathbf{A}_{k}\mathbf{B}^{T}_{k}}$, where $\mathbf{A}_{k}$ and $\mathbf{B}_{k}^{T}$ are the $k$th column of $\mathbf{A}$ and $k$th row of $\mathbf{B}$ respectively. The proposed scheme is based on the following steps.

• •

Step 1. The master node forms the following polynomial matrices based on column-blocks of $\mathbf{A}$ and row-blocks of $\mathbf{B}$ as follows

[TABLE]

and

[TABLE]

Then $\mathbf{C}=\mathbf{A}\mathbf{B}$ can be recovered from $\hat{\mathbf{C}}(x)=\hat{\mathbf{A}}(x)\hat{\mathbf{B}}(x)^{T}$ if we have the value of $\hat{\mathbf{C}}(x)$ for seven distinct values of $x\in\mathbb{R}$. More precisely, each entry of $\hat{\mathbf{C}}(x)$ is a polynomial of degree six. Let us focus on block $(k,\ell)$ of $\hat{\mathbf{C}}(x)$ denoted by $\hat{\mathbf{C}}_{k,\ell}(x)$. Then

[TABLE]

In this expansion, one can verify that $\hat{\mathbf{C}}_{3}^{(k,\ell)}=\mathbf{C}_{k,\ell}$. If we have the value of $\hat{\mathbf{C}}_{k,\ell}(x)$ for seven distinct values of $x\in\mathbb{R}$ then all the coefficients of $\hat{\mathbf{C}}_{k,\ell}(x)$ can be calculated using polynomial interpolation. In particular, $\mathbf{C}_{k,\ell}$, which is the coefficient of $x^{3}$, can be calculated.

• •

Step 2. To reduce the dimension of this product, the count-sketch method is used. Assume we construct three count-sketches for $\hat{\mathbf{A}}(x)$. Let us assume that the sketch polynomials of the row-blocks of $\hat{\mathbf{A}}(x)$ are described as:

[TABLE]

where $\mathbf{F}_{i}$ is the sketch polynomial of matrix $\hat{\mathbf{A}}(x)$ using the hash function $h_{i,\hat{\mathbf{A}}}:[0:3]\to[0:1]$. In particular, the hash functions are considered as follows

[TABLE]

Same as before, assume we have three count-sketches for $\hat{\mathbf{B}}(x)$. To be specific, assume that the related sketch polynomials are defined as follows

[TABLE]

where $\mathbf{G}_{i}$ is the sketch polynomial of matrix $\hat{\mathbf{B}}(x)$ using the hash function $h_{i,\hat{\mathbf{B}}}:[0:3]\to[0:1]$ as follows

[TABLE]

We note that the ${\mathbf{F}_{\ell}}(x,\alpha){\mathbf{G}_{\ell}}(x,\alpha)^{T}$ can be considered as a sketch polynomial for $\hat{\mathbf{C}}(x)$, where $\ell=1,2,3$. For example, ${\mathbf{F}_{1}}(x,\alpha){\mathbf{G}_{1}}(x,\alpha)^{T}$ can be written as

[TABLE]

where in this expansion,

[TABLE]

Each entry of $\hat{\mathbf{C}}(x)$ is a polynomial of degree six in which the coefficient of $x^{3}$ is the combination of entries of original result $\mathbf{C}$. This can be explained better as follows

[TABLE]

Then according to (28) and (29), and the discussion followed we have

[TABLE]

Particularly in the expansion (47), the terms in (48)–(50) are of interest. The reason the other coefficients are not interesting is that the coefficients that we are looking for appear only in these terms. Thus, we have another count-sketch hidden in the count-sketch of $\hat{\mathbf{C}}(x)$. These three coefficients (48)–(50) form a count-sketch of the blocks of $\mathbf{C}$. The reason that the special coefficients of (47) form a count-sketch of the blocks of $\mathbf{C}$, is the structure used in this scheme. This structure, which we name CodedSketch, is the concatenation of the count-sketch and the entangled polynomial code. Other ${\mathbf{F}_{\ell}}(x,\alpha){\mathbf{G}_{\ell}}(x,\alpha)^{T}$ form independent count-sketch similarly.

In the following, the computation of these sketch polynomials over a distributed system is proposed and to form the results of ${\mathbf{F}_{\ell}}(x,\alpha){\mathbf{G}_{\ell}}(x,\alpha)^{T}$ where $\ell=1,2,3$ efficiently, we use another layer of structured code motivated by improved entangled polynomial codes [9].

• •

Step 3. The master node creates the following polynomials and encodes the $\mathbf{F}_{\ell}(x,\alpha)$ and $\mathbf{G}_{\ell}(x,\alpha)$

[TABLE]

where $\beta_{1},\beta_{2},\beta_{3}\in\mathbb{R}$ are three distinct values. These polynomials are linear combinations of sketch polynomials created using different hash functions. It can be seen that $\mathbf{F}(x,\alpha,\beta_{1})=\mathbf{F}_{1}(x,\alpha)$, $\mathbf{F}(x,\alpha,\beta_{2})=\mathbf{F}_{2}(x,\alpha)$ and $\mathbf{F}(x,\alpha,\beta_{3})=\mathbf{F}_{3}(x,\alpha)$ and so does $\mathbf{G}(x,\alpha,\omega)$. Since the extraction of hidden count-sketch of the blocks of $\mathbf{C}$ is desired, we choose $\alpha=x^{4}$ and $\omega=x^{15}$. Let $\mathbf{F}(x)\triangleq\mathbf{F}(x,x^{4},x^{15})$ and $\mathbf{G}(x)\triangleq\mathbf{G}(x,x^{4},x^{15})$. The number $\theta_{j}\in\mathbb{R}$ is dedicated to the $j$th worker node, where $\theta_{i}\neq\theta_{j}$ if $i\neq j$. Therefore, the master node sends $\mathbf{F}(\theta_{j})\in\mathbb{R}^{\frac{r}{4}\times\frac{s}{4}}$ and $\mathbf{G}(\theta_{j})\in\mathbb{R}^{\frac{s}{4}\times\frac{t}{4}}$ to $j$th worker node.

• •

Step 4. Having received matrices $\mathbf{F}(\theta_{j})$ and $\mathbf{G}(\theta_{j})$ from master node, the $j$th worker node multiplies these two matrices. Then it returns the result, i.e., $\mathbf{F}(\theta_{j})\mathbf{G}(\theta_{j})^{T}$ to the master node. The result calculated at node $j$ can be written as

[TABLE]

By substituting the $\mathbf{F}_{\ell}$ and $\mathbf{G}_{k}$ in (53), where $\ell,k=1,2,3$, the polynomial with $75$ coefficients is created in which the count-sketch results of the blocks of $\mathbf{C}$ are located.

• •

Step 5. The master node can recover all of the polynomials’ coefficients by receiving the computation results of any 75 worker nodes. That is because the recovering process is equivalent to interpolating a polynomial of degree 74 given its value at 75 points. After interpolation and recovering the coefficients, a polynomial of degree 74 is created. Assume that, in this polynomial, all $x^{15}$ are replaced by $\omega$. In this case, a bivariate polynomial is achieved in which can we choose $\omega=\beta_{1},\beta_{2},\beta_{3}$ to calculate three sketch polynomials $\mathbf{F}_{1}(x)\mathbf{G}_{1}(x)^{T}$, $\mathbf{F}_{2}(x)\mathbf{G}_{2}(x)^{T}$ and $\mathbf{F}_{3}(x)\mathbf{G}_{3}(x)^{T}$ for $\hat{\mathbf{C}}(x)$ respectively. According to (48) we need the coefficients of $x^{3}$, $x^{7}$ and $x^{11}$ to find the hidden count-sketches of the blocks of $\mathbf{C}$ in the sketch polynomial of $\hat{\mathbf{C}}(x)$.

• •

Step 6. The coefficients $x^{3}$, $x^{7}$ and $x^{11}$ of the three sketch polynomials $\mathbf{F}_{1}(x)\mathbf{G}_{1}(x)^{T}$, $\mathbf{F}_{2}(x)\mathbf{G}_{2}(x)^{T}$ and $\mathbf{F}_{3}(x)\mathbf{G}_{3}(x)^{T}$ are shown in Table I. To achieve an approximation of ${\mathbf{C}}$ more accurately, the master node takes the median of these estimations after multiplying them to the corresponding sign functions. For example, to approximate the value of $[\mathbf{C}_{2,0}]_{i,j}$ for all $i\in[0:\frac{r}{m}-1]$ and $j\in[0:\frac{t}{n}-1]$, the master node does the following

[TABLE]

In another form the master node takes the median of the following terms

[TABLE]

Therefore, in this scheme, the master node sends two matrices to each worker and each worker sends back the result of multiplication of these matrices to the master node. If a sufficient subset of worker nodes return their results to the master, finally the master node can recover an approximation of $\mathbf{C}$ by using polynomial interpolation and median of the desired coefficients.

IV-B General CodedSketch Design

Now we present the proposed scheme for distributed computation of an approximation for the matrix multiplication using CodedSketch in a general setting. In summary, the encoding step of CodedSketch has three important parts which use the ideas of entangled polynomial codes and count-sketch to form the useful combination of randomness and structured codes such that an approximated version of multiplication is computed. First of all, the input matrices $\mathbf{A}$ and $\mathbf{B}$ are partitioned into equal-size $m\times p$ and equal-size $p\times n$ sub-matrices as (19) and (20). In this setting, the following steps will be taken:

• •

Step 1: The master node forms the following polynomial coded matrices based on column-blocks of $\mathbf{A}$ and row-blocks of $\mathbf{B}$ as follows

[TABLE]

where $\mathbf{A}_{k}$ and $\mathbf{B}_{k}^{T}$ are the $k$th column of $\mathbf{A}$ and $k$th row of $\mathbf{B}$, for $k=0,\ldots,p-1$. By this coding, one can verify that the block $(k,\ell)$ of $\hat{\mathbf{C}}(x)=\hat{\mathbf{A}}(x)\hat{\mathbf{B}}(x)^{T}$ is a polynomial of degree $(2p-2)$, where the coefficient of $x^{p-1}$ is the block $(k,\ell)$ of $\mathbf{C}$. More precisely, the block $(k,\ell)$ of $\hat{\mathbf{C}}(x)$ can be written as

[TABLE]

where in particular $\hat{\mathbf{C}}_{p-1}^{(k,\ell)}=\mathbf{C}_{k,\ell}$, and $\hat{\mathbf{C}}_{\nu}^{(k,\ell)}$, $\nu=0,\ldots,2p-2$, $\nu\neq p-1$ are some real matrices with the same size as $\hat{\mathbf{C}}_{p-1}^{(k,\ell)}$. This means that if we have the value of $\hat{\mathbf{C}}(x)$ for $2p-1$ distinct values of $x\in\mathbb{R}$, we can recover $\hat{\mathbf{C}}(x)$, and in particular the result $\mathbf{C}$. However, our goal is not to compute $\mathbf{C}$ exactly. In the following, we use count-sketch method, to compute the approximation of $\mathbf{C}$, and in return reduce the recovery threshold.

• •

Step 2: In this step, we use the hash functions $h_{i}:\mathcal{U}\to[0:b^{\prime}-1]$ and $\tilde{h}_{i}:\mathcal{U}\to[0:b^{\prime}-1]$, chosen uniformly at random from the family of 2-wise independent hash functions to sketch the row-blocks and the column-blocks of coded matrices $\hat{\mathbf{A}}(x)$ and $\hat{\mathbf{B}}(x)^{T}$ respectively, for $i=1,...,d$, and for some $b^{\prime}\in\mathbb{N}$. To sketch, we also need $s_{\ell}:\mathcal{U}\to\{-1,1\}$ and $\tilde{s}_{\ell}:\mathcal{U}\to\{-1,1\}$, for $\ell=1,\ldots,d$, which are chosen uniformly at random from the family of pairwise–independent sign functions. Let $\mathbf{F}_{\ell}(x,\alpha)$ and $\mathbf{G}_{\ell}(x,\alpha)^{T}$ be the sketch polynomials of the row-blocks of $\hat{\mathbf{A}}(x)$ based on $h_{\ell}$ and $s_{\ell}$, and the column-blocks of $\hat{\mathbf{B}}(x)^{T}$ based on $\tilde{h}_{\ell}$ and $\tilde{s}_{\ell}$, for $\ell=1,\ldots,d$. More precisely,

[TABLE]

where $\hat{\mathbf{A}}_{i,0}(x)$ and $\hat{\mathbf{B}}_{0,i}(x)$ are the $i$th block element of $\hat{\mathbf{A}}(x)$ and $\hat{\mathbf{B}}(x)$.

• •

Step 3: The master node creates the following polynomials using improved entangled polynomial codes

[TABLE]

for some variables $\alpha$ and $\omega$, where $\beta_{1},\dots,\beta_{d}\in\mathbb{R}$ are some distinct values.

The master node chooses $\alpha=x^{p}$ and $\omega=x^{2pb^{\prime}-1}$. Let $\mathbf{F}(x)\buildrel\Delta\over{=}\mathbf{F}(x,x^{p},x^{2pb^{\prime}-1})$ and $\mathbf{G}(x)\buildrel\Delta\over{=}\mathbf{G}(x,x^{p},x^{2pb^{\prime}-1})$. Then, the master node sends $\mathbf{F}(\theta_{j})$ and $\mathbf{G}(\theta_{j})$ matrices to the $j$th worker node, where $\theta_{j}$ is an arbitrary element in $\mathbb{R}$ and $\theta_{i}\neq\theta_{j}$ if $i\neq j$. The size of the transmitted matrices are $\frac{1}{pm}$ fraction of $\mathbf{A}$ and $\frac{1}{pn}$ fraction of $\mathbf{B}$ respectively. In other words, $\mathbf{F}(\theta_{j})\in\mathbb{R}^{\frac{r}{m}\times\frac{s}{p}}$ and $\mathbf{G}(\theta_{j})\in\mathbb{R}^{\frac{s}{p}\times\frac{t}{n}}$.

• •

Step 4: Worker node $j$ has $\mathbf{F}(\theta_{j})$ and $\mathbf{G}(\theta_{j})$ and computes $\mathbf{P}({\theta_{j}})\buildrel\Delta\over{=}\mathbf{F}(\theta_{j})\mathbf{G}(\theta_{j})^{T}$, and sends the result to the master node.

• •

Step 5: The master node waits to receive answers from $(2pb^{\prime}-1)(2d-1)$ worker nodes. It can be verified that the polynomial

[TABLE]

has degree of $(2pb^{\prime}-1)(2d-1)-1$, where $\omega=x^{2pb^{\prime}-1}$. Thus, the master node can interpolate $\mathbf{P}(x)$, having access to $\mathbf{P}(\theta_{j})$, for $(2pb^{\prime}-1)(2d-1)$ distinct values of $\theta_{j}$.

Assume that, in the polynomial $\mathbf{P}(x)$, the master node replaces $x^{2pb^{\prime}-1}$ by variable $\omega$. By this substitution a bivariate polynomial of $x$ and $\omega$ is formed. To recover $\mathbf{F}_{\eta}(x)\mathbf{G}_{\eta}(x)^{T}$, for any $\eta\in\{1,\ldots,d\}$, the master node then replaces the variable $\omega$ with $\beta_{\eta}$.

In Appendix -A, we show that in $\mathbf{F}_{\eta}(x)\mathbf{G}_{\eta}(x)^{T}$, the coefficients of $x^{kp+p-1}$ for $k=0,...,2b^{\prime}-2$, denoted by $\mathbf{P}_{k,p-1}^{(\eta,\eta)}$, are equal to

[TABLE]

where $\mathbf{C}_{i,j}$ is the block $(i,j)$ of $\mathbf{C}$.

• •

Step 6: Recall that the goal is to approximate the original result of multiplication, i.e., $\mathbf{C}$.

[TABLE]

For each $[\tilde{\mathbf{C}}_{k,\ell}]_{i,j}$, the recovering phase is done as follows

[TABLE]

where $i\in[0:\frac{r}{m}-1]$, $j\in[0:\frac{t}{n}-1]$, $k\in[0,m-1]$ and $\ell\in[0,n-1]$.

Proof of Theorem 4.

We note that each count-sketch used for $\mathbf{A}$ and $\mathbf{B}$ in (59) and (60) has length $b^{\prime}$, while each resulting count-sketch in (64) for $\mathbf{C}$ has length $b=2b^{\prime}-1$.

According to Theorem 1 in Section II on preliminaries, to have an unbiased, $(\epsilon,\delta)$-accurate approximation of C, it is sufficient to choose $d\geq\log\frac{1}{\delta}$ and $b^{\prime}\geq\frac{3}{\epsilon^{2}}$. On the other hand, from (63), the degree of the polynomial $\mathbf{P}(x)$ is equal to \big{(}(2pb^{\prime}-1)(2d-1)-1\big{)}. Thus, to recover $\mathbf{P}(x)$, the master node needs $\mathbf{P}(\theta_{j})$ for \big{(}(2pb^{\prime}-1)(2d-1)\big{)} distinct $\theta_{j}$’s. Thus the recovery threshold of the scheme is the minimum of \big{(}(2pb^{\prime}-1)(2d-1)\big{)} subject to $d\geq\log\frac{1}{\delta}$, $b^{\prime}\geq\frac{3}{\epsilon^{2}}$, and $d,b^{\prime}\in\mathbb{N}$, as follows

[TABLE]

∎

Proof of Theorem 5.

According to Theorem 3 in Section II on preliminaries, if we choose the length of the count-sketches as $b^{\prime}=\mathcal{O}(\frac{\kappa}{\epsilon^{2}})$ and the number of count-sketches as $d$, then we have the following guarantee

[TABLE]

where $\mathrm{err}_{2}^{(\kappa,(i,j))}(\mathbf{C})={(\sum_{(k,\ell)\notin\mathcal{T}_{\kappa}}{|[\mathbf{C}_{k,\ell}]_{i,j}|^{2}})}^{\frac{1}{2}}$ where $\mathcal{T}_{\kappa}$ is the set of indices of the blocks of $\mathbf{C}$ whose entry $(i,j)$ is in the $\kappa$-largest entry $(i,j)$ among all blocks. Also, $i\in[0:\frac{r}{m}-1]$, $j\in[0:\frac{t}{n}-1]$, $k\in[0:m-1]$ and $\ell\in[0:n-1]$. From (68) we have the following guarantee

[TABLE]

where $i^{\prime}=i\mod\frac{r}{m}$, $j^{\prime}=j\mod\frac{t}{n}$, and $[\mathbf{C}]_{i,j}$ is the $(i,j)$th entry of $\mathbf{C}$.

Recall that $\mathrm{err}_{2}^{(\kappa,(i^{\prime},j^{\prime}))}(\mathbf{C})={(\sum_{(k,\ell)\notin\tilde{\mathcal{T}}_{\kappa}}{|[\mathbf{C}_{k,\ell}]_{i^{\prime},j^{\prime}}|^{2}})}^{\frac{1}{2}}$, where $\tilde{\mathcal{T}}_{\kappa}$ is the set of indices of the blocks of $\mathbf{C}$ whose entry $(i^{\prime},j^{\prime})$ is in the $\kappa$-largest entry $(i^{\prime},j^{\prime})$ among all blocks. Thus, according to Remark 3, and the result of Theorem 4, it is sufficient to choose $b^{\prime}\geq\frac{3\kappa}{\epsilon^{2}}$ and $d\geq\log(mn)$, where $mn$ is the number of block entries of $\mathbf{C}$. So, if the result of multiplication is $\kappa$-sparse, the CodedSketch method computes $\mathbf{C}$ with probability $1-\delta=1-\frac{1}{(mn)^{\Theta(1)}}$ by choosing parameters properly. So, we have

[TABLE]

∎

IV-C * Complexity Analysis*

Encoding Complexity: In CodedSketch scheme, the master node in the encoding step performs three steps:

1- The master node forms the polynomial matrices using entangled polynomial codes based on the column-blocks of $\mathbf{A}$ and the row-blocks5 of $\mathbf{B}$. In other words, the master node forms two polynomial matrices in which the columns of the partitioned matrix $\mathbf{A}$ and the rows of the partitioned matrix $\mathbf{B}$ are their coefficients respectively.

2- The master node constructs sketch polynomials. The computational complexity of creating a sketch polynomial for a vector with $n$ entries is $\mathcal{O}(n)$. So, the computational complexity of constructing $d$ sketch polynomials for $\hat{\mathbf{A}}(x)$ and $\hat{\mathbf{B}}(x)^{T}$ is $\mathcal{O}(dm+dn)$. Considering the size of each block in the partitioned matrices, the overall computational complexity in this step is $\mathcal{O}(dm\frac{r}{m}\frac{s}{p}+dn\frac{s}{p}\frac{t}{n})$.

3- The master node encodes $d$ sketch polynomials and creates two polynomials $\mathbf{F}(x)$ and $\mathbf{G}(x)$ using improved entangled polynomial codes.

4- The master node evaluates two polynomials in $N$ distinct values. Some efficient polynomial algorithms such as Horner’s method [32] can be used to evaluate a polynomial of degree $n$ in one point with the complexity of $\mathcal{O}(n)$. In this phase of the algorithm, we need to compute two polynomials of degree $(2pdb^{\prime}-pb^{\prime}-d)$, where the coefficients are matrices of size $\frac{rs}{mp}$ and $\frac{st}{pn}$, at $N$ points. Thus, this step requires a complexity of $\mathcal{O}(Nsdb^{\prime}(\frac{r}{m}+\frac{t}{n}))$.

Therefore, the complexity of the encoding process of CodedSketch is $\mathcal{O}(\frac{sd}{p}(r+t)+Nsdb^{\prime}(\frac{r}{m}+\frac{t}{n}))$, where $\frac{2b^{\prime}}{mn}\leq 1$, according to the properties of count-sketch method.

Decoding Complexity: The computation performed by the master node in the decoding step includes interpolation of a matrix-polynomial of degree $((2pb^{\prime}-1)(2d-1)-1)$, where the coefficients are matrices of size $\frac{rt}{mn}$. The complexity of interpolation of a polynomial of degree $k-1$ is $\mathcal{O}(k\log^{2}k)$ [33, 34]. Thus complexity of decoding in this scheme is $\mathcal{O}(2prtd\frac{2b^{\prime}}{mn}\log^{2}(4pb^{\prime}d))$, where $\frac{2b^{\prime}}{mn}\leq 1$. This computation complexity is a linear function of the input size. Using this interpolation, the count-sketches of $\mathbf{C}$ are obtained. The median of these count-sketches for $rt$ times is returned as an approximation of $\mathbf{C}$ in the last step, which has a complexity of $O(rtd)$ on average [35].

Computational Complexity at Each Worker Node: In this setting, each worker node computes only the multiplication of two matrices of dimensions $\frac{r}{m}\times\frac{s}{p}$ and $\frac{s}{p}\times\frac{t}{n}$.

Communication Load: The master node only sends two matrices of dimensions $\frac{r}{m}\times\frac{s}{p}$ and $\frac{s}{p}\times\frac{t}{n}$ to each worker node and it receives one matrix of size $\frac{r}{m}\times\frac{t}{n}$ from worker nodes. Recall that, using CodedSketch the approximation of matrix multiplication is computed in one iteration.

Remark 10*:*

In [14], the idea of OverSketch for matrix multiplication is proposed in which some extra count-sketches are used, as a redundancy, to mitigate the effect of stragglers. In particular, consider two input matrices $\mathbf{A}\in\mathbb{R}^{r\times s}$ and $\mathbf{B}\in\mathbb{R}^{s\times t}$. The count-sketches of two input matrices are computed using sketch matrix $\mathbf{S}\in\mathbb{R}^{s\times z}$, as $\check{\mathbf{A}}=\mathbf{A}\mathbf{S}$ and $\check{\mathbf{B}}=\mathbf{S}^{T}\mathbf{B}$. Then the block sub-matrices are created by partitioning $\check{\mathbf{A}}$ into $\frac{rz}{q^{2}}$ sub-matrices and $\check{\mathbf{B}}$ into $\frac{zt}{q^{2}}$ sub-matrices of size $q\times q$. Accordingly, matrix $\mathbf{C}=\mathbf{A}\mathbf{B}$ is partitioned into $\frac{rt}{q^{2}}$ sub-matrices of size $q\times q$. Then each block of matrix $\mathbf{C}$ is approximately calculated by multiplying the corresponding row-block of $\check{\mathbf{A}}$ and column-block of $\check{\mathbf{B}}$. In that scheme, the set of worker nodes is partitioned into $\eta$ subsets as $\mathcal{W}=\{\mathcal{W}_{1},\dots,\mathcal{W}_{\frac{rt}{q^{2}}}\}$. Each subset of worker nodes computes one block of $\mathbf{C}$ approximately, where each worker node multiplies one sub-matrix in the row-block of $\check{\mathbf{A}}$ into the corresponding sub-matrix in the column-block of $\check{\mathbf{B}}$ and then the results are added. In OverSketch, it is set $z=z^{*}$ for some $z^{*}\in\mathbb{N}$, such that each partition $\mathcal{W}_{i}$ with at least $\frac{z^{*}}{q}$ worker nodes can approximately calculate each block of ${\mathbf{C}}$ within the following accuracy

[TABLE]

where ${\tilde{\mathbf{C}}_{\text{os}}}$ is the approximation of $\mathbf{C}$ using OverSketch. In [14], it is shown that $z^{*}=\frac{2}{\xi\theta}$. We note that each partition of worker nodes is dedicated to approximately calculate one block of $\mathbf{C}$, and does not help in calculating other blocks of $\mathbf{C}$.

In OverSketch, if we want the system to be resistant against $e$ stragglers for some $e\in\mathbb{N}$, then we increase the number of worker nodes in each partition from $\frac{z^{*}}{q}$ to $\frac{z^{*}}{q}+e$. Therefore, $z=z^{*}+eq$. As a result, to calculate all blocks of ${\mathbf{C}}$, we need

[TABLE]

total worker nodes in OverSketch, where $m=\frac{r}{q}$, $n=\frac{t}{q}$ and $z^{*}=\frac{2}{\xi\theta}$.

In CodedSketch, we combine the coding theory and randomized approximation algorithms to approximately calculate the matrix multiplication. In the proposed scheme, $d$ count-sketches of the blocks of matrix $\mathbf{C}$ will be formed after the decoding step, and the master node will recover an approximated version of $\mathbf{C}$ from these count-sketches. In CodedSketch, according to Theorem 1, if we choose the length of the count-sketches as $b^{\prime}=\mathcal{O}(\frac{1}{\epsilon^{2}})$ and the number of count-sketches as $d=\mathcal{O}(1/\delta)$, then we have (17) as the precision guarantee.

In CodedSketch, according to (63), the total number of worker nodes is computed as follows

[TABLE]

Now, we compare OverSketch and CodedSketch in terms of recovery threshold and in terms of approximation precision bounds as follows.

Comparison in terms of recovery threshold: Here we compute the recovery threshold of OverSketch, $N_{\textrm{os}}^{th}$. Assume that out of $N_{\textrm{os}}$ in (71), $e+1$ of them do not respond. Then it may happen that, this $e+1$ worker nodes are from one partition of worker nodes assigned to compute one specific block. Then, there are only $\frac{z^{*}}{q}-1$ worker nodes left for that partition, which are not enough to calculate that block within the required accuracy. Therefore, the recovery threshold of OverSketch can be written as

[TABLE]

By eliminating $e$ from (72) and (74), the recovery threshold is equal to

[TABLE]

Note that $N_{\textrm{os}}^{th}$ is a function of $N_{\textrm{os}}$, and the reason is that in OverSketch, the gain of coding is not exploited in calculating different blocks. This is in contrary with what we have in CodedSketch. In CodedSketch, calculation in each server can help the calculation of all blocks, as we expect from coding. In CodedSketch method the approximated result can be calculated if any

[TABLE]

subset of worker nodes respond.

Now, we compare OverSketch and CodedSketch with the same approximation precision bounds.

Comparison in terms of computation precision: Inequalities in (71) and (17) do not provide the precision guarantee with the same level and can not be compared. To resolve this issue, we take the following steps.

In CodedSketch, we have (17) as the precision guarantee. From (17) we have

[TABLE]

where $i^{\prime}_{m}=i_{m}\mod\frac{r}{m}$, $j^{\prime}_{m}=j_{m}\mod\frac{t}{n}$, and (i_{m},j_{m})={\arg\max}_{i,j}\big{\{}|[\mathbf{C}]_{i,j}-[\tilde{\mathbf{C}}]_{i,j}|\big{\}}. To be able to compare OverSketch and CodedSketch, we loosen bound (77) to form the following bound

[TABLE]

According to (71) and (78), to compare the total number of worker nodes in these two schemes, let

[TABLE]

From (72) and (73), the total number of worker nodes in two schemes are computed as follows

[TABLE]

where $\theta=\delta$ and ${{\xi=\frac{rt\epsilon^{2}||\mathbf{C}||^{2}_{F,(i^{\prime}_{m},j^{\prime}_{m})}}{||\mathbf{A}||_{F}^{2}||\mathbf{B}||_{F}^{2}}}}$ in (81).

Now we are ready to compare these two schemes under the same level of computation precision. Figure 2 shows the comparison between the total number of worker nodes versus the number of stragglers in CodedSketch and OverSketch. In this simulation, for Fig. 2(a) we generate two random matrices $\mathbf{A}\in\mathbb{R}^{700\times 100}$ and $\mathbf{B}\in\mathbb{R}^{100\times 700}$ with the elements chosen uniformly at random on the $[-100,100]$. Also, we choose $m=n=70$, $p=10$ and $\epsilon=0.1$, $\delta=0.02$. Also, for Fig. 2(b) we generate two random matrices $\mathbf{A}\in\mathbb{R}^{500\times 50}$ and $\mathbf{B}\in\mathbb{R}^{50\times 500}$ with the elements chosen uniformly at random on the $[-100,100]$. Also, we choose $m=n=50$, $p=5$ and $\epsilon=0.1$, $\delta=0.2$. Figure 2 shows that compared to OverSketch, in some cases, CodedSketch can reduce the total number of worker nodes by order of magnitudes with the same approximation precision bound. It is worth noting that to make the precision guarantee bound similar for two schemes, we took some steps to form (78), which is a very loose bound of CodedSketch but still, we can see that these two schemes are comparable.

V Conclusion

In this paper, we proposed a coded sketch scheme to approximately compute the multiplication of two large-scale matrices in a distributed straggler-resistant system. The proposed scheme exploits the structure of matrices in multiplication, and still uses the randomized pre-compression scheme based on sketching schemes. This scheme provides an upper-bound on the recovery threshold as a function of the required accuracy of computation and the probability that the required accuracy can be violated. In addition, it provides an upper-bound on the recovery threshold for the case that the result of the multiplication is sparse, and the exact result is required. The next step would be to consider more general functions, specially polynomial functions.

VI Acknowledgment

This research was supported by Iran National Science Foundation (INSF) under contract No. 98019168.

-A *Proof of (64) *

In (61) and (62) it can be seen that $\mathbf{F}(x,\alpha,\eta)=\mathbf{F}_{\eta}(x,\alpha)$ and $\mathbf{G}(x,\alpha,\eta)=\mathbf{G}_{\eta}(x,\alpha)$ for $\eta=1,\dots,d$, and we know that each $\mathbf{F}_{\eta}(x,\alpha)\mathbf{G}_{\eta}(x,\alpha)^{T}$ is a sketch polynomial for $\hat{\mathbf{C}}(x)$, i.e.,

[TABLE]

Also, (83) can be written as follows

[TABLE]

where in this expansion,

[TABLE]

As mentioned before, the block $(i,j)$ of $\hat{\mathbf{C}}(x)$, i.e., $\hat{\mathbf{C}}_{i,j}(x)$ is a polynomial of degree $(2p-2)$ where we can recover the linear combination of some blocks of $\mathbf{C}$ as the coefficient of $x^{p-1}$. So, in the sketch polynomial of $\hat{\mathbf{C}}(x)$ there is a hidden count-sketch for blocks of $\mathbf{C}$. Also, (84) can be explained in the following form

[TABLE]

where the linear combinations of some blocks of $\mathbf{C}$ are located in $\mathbf{P}_{k,p-1}^{(\eta,\eta)}$ for $k=0,..,2b^{\prime}-2$. Thus, according to (85), and (86), we have

[TABLE]

where $\mathbf{C}_{i,j}$ is the block $(i,j)$ of $\mathbf{C}$.