Gradient Coding Based on Block Designs for Mitigating Adversarial Stragglers

TL;DR

This paper introduces a new class of approximate gradient codes based on block designs, specifically BIBDs, to effectively mitigate adversarial stragglers in distributed gradient computations, ensuring robustness and efficient decoding.

Contribution

The paper proposes a novel gradient coding scheme using balanced incomplete block designs that are resilient to adversarial stragglers and characterizes the fundamental limits of such adversarial attacks.

Findings

Approximate gradient codes based on BIBDs depend only on the number of stragglers.

Codes based on symmetric BIBDs maximize the adversarial threshold.

Proposed codes enable computationally efficient decoding at the server.

Abstract

Distributed implementations of gradient-based methods, wherein a server distributes gradient computations across worker machines, suffer from slow running machines, called 'stragglers'. Gradient coding is a coding-theoretic framework to mitigate stragglers by enabling the server to recover the gradient sum in the presence of stragglers. 'Approximate gradient codes' are variants of gradient codes that reduce computation and storage overhead per worker by allowing the server to approximately reconstruct the gradient sum. In this work, our goal is to construct approximate gradient codes that are resilient to stragglers selected by a computationally unbounded adversary. Our motivation for constructing codes to mitigate adversarial stragglers stems from the challenge of tackling stragglers in massive-scale elastic and serverless systems, wherein it is difficult to statistically model stragglers. Towards this end, we propose a class of approximate gradient codes based on balanced incomplete block designs (BIBDs). We show that the approximation error for these codes depends only on the number of stragglers, and thus, adversarial straggler selection has no advantage over random selection. In addition, the proposed codes admit computationally efficient decoding at the server. Next, to characterize fundamental limits of adversarial straggling, we consider the notion of 'adversarial threshold' -- the smallest number of workers that an adversary must straggle to inflict certain approximation error. We compute a lower bound on the adversarial threshold, and show that codes based on symmetric BIBDs maximize this lower bound among a wide class of codes, making them excellent candidates for mitigating adversarial stragglers.

Tables

Table 1. TABLE I: Notation for Gradient Coding

$N$	Number of workers
$K$	Number of data partitions
$L$	Computational load per worker
$R$	Replication factor
$𝑬$	Encoding matrix of size $K\times N$

Table 2. TABLE II: Notation for BIBDs

$v$	Number of points
$b$	Number of blocks
$k$	Number of points per block
$r$	Number of blocks containing a point
$\lambda$	Number of blocks containing a pair of points
$𝑴$	Incidence matrix of size $v\times b$

Table 3. TABLE III: Gradient Codes Using Block Designs. The parameter q 𝑞 q is a power of a prime and m ≥ 2 𝑚 2 m\geq 2 is an integer.

Class	Design	Parameters of GC
Symmetric BIBD	Projective Geometry (PG)	$N=K=\frac{\left(q^{m+1}-1\right)}{\left(q-1\right)}$, $L=R=\frac{\left(q^m-1\right)}{\left(q-1\right)}$
Dual of a BIBD	Dual of Affine Geometry (Dual AG)	$N=q^m$, $K=q\frac{\left(q^m-1\right)}{\left(q-1\right)}$, $L=\frac{\left(q^m-1\right)}{\left(q-1\right)}$, $R=q^{m-1}$
Resolvable BIBD	Affine Geometry (AG)	$N=q\frac{\left(q^m-1\right)}{\left(q-1\right)}$, $K=q^m$, $L=q^{m-1}$, $R=\frac{\left(q^m-1\right)}{\left(q-1\right)}$

Equations

$$\underset{\mathbf{w}}{\min }\frac{1}{M}\sum _{i=1}^M\ell \left({\mathbf{x}}_i,y_i;\mathbf{w}\right),$$

$${\mathrm{err}}_{\mathcal{F}}\left(\mathbf{E}\right)=\underset{\mathbf{v}\in {\mathbb{R}}^{N-S}}{\min }{\Vert {\mathbf{E}}_{\mathcal{F}}\mathbf{v}-1_K\Vert }_2^2,$$

$${\mathrm{err}}_S\left(\mathbf{E}\right)=\underset{\begin{array}{c}\mathcal{F}\subset [N]\\ |\mathcal{F}|=N-S\end{array}}{\max }\underset{\mathbf{v}\in {\mathbb{R}}^{N-S}}{\min }{\Vert {\mathbf{E}}_{\mathcal{F}}\mathbf{v}-1_K\Vert }_2^2.$$

$${\mathbf{v}}_{\mathrm{opt}}=\left(\frac{L}{L+\lambda (N-S-1)}\right)1_{N-S}.$$

$${\mathrm{err}}_S\left(\mathbf{E}\right)=K-\frac{L^2(N-S)}{L+\lambda (N-S-1)}.$$

$${\mathbf{v}}_{\mathrm{opt}}(j)=\{\begin{array}{ll}1& \text{for}{\hat{S}}_{i-1}+1\le j\le {\hat{S}}_i\\ 0& \text{otherwise},\end{array}$$

$${\mathbf{v}}_{\mathrm{opt}}(j)=\frac{L/\left(L-\mu (K/L-S_i)\right)}{\left(1+{\sum }_{p=1}^R\frac{\mu (K/L-S_p)}{L-\mu (K/L-S_p)}\right)}$$

$${\mathrm{err}}_{\mathcal{F}}\left(\mathbf{E}\right)$$

$${\mathrm{err}}_S\left({\mathbf{E}}_{\text{PG}}\right)=\frac{S}{(q+1)+\frac{q+1-S}{q}}$$

$$S^{\ast }(\eta ):=\arg \underset{\begin{array}{c}\mathcal{T}\subset \mathcal{D}\\ |\mathcal{T}|=\eta \end{array}}{\min }|\mathcal{N}\left(\mathcal{T}\right)|.$$

$$S^{\ast }(\eta )\ge \left(\frac{3L-{\lambda }_2}{L+{\lambda }_2}\right)\eta =:S_{LB}^{\ast }(\eta ),$$

$$(\mathbf{G}+\mathbf{H}{)}^{-1}={\mathbf{G}}^{-1}-\frac{1}{1+\mathrm{tr}\left(\mathbf{H}{\mathbf{G}}^{-1}\right)}{\mathbf{G}}^{-1}\mathbf{H}{\mathbf{G}}^{-1}.$$

$${\mathbf{v}}_{\mathrm{opt}}=\arg \underset{\mathbf{v}\in {\mathbb{R}}^{N-S}}{\min }{\Vert {\mathbf{E}}_{\mathcal{F}}\mathbf{v}-1_K\Vert }_2^2.$$

$${\mathbf{E}}_{\mathcal{F}}^T{1}_K$$

$${\mathbf{E}}_{\mathcal{F}}^T{\mathbf{E}}_{\mathcal{F}}$$

$$({\mathbf{E}}_{\mathcal{F}}^T{\mathbf{E}}_{\mathcal{F}}{)}^{-1}$$

$${\mathbf{v}}_{\mathrm{opt}}$$

$${\mathrm{err}}_{\mathcal{F}}\left(\mathbf{E}\right)$$

$${\mathbf{v}}_{\mathrm{opt}}=\arg \underset{\mathbf{v}\in {\mathbb{R}}^{N-S}}{\min }{\Vert {\mathbf{E}}_{\mathcal{F}}\mathbf{v}-1_K\Vert }_2^2.$$

$${\mathbf{E}}_{\mathcal{F}}^T{1}_K$$

$${\mathbf{E}}_{\mathcal{F}}^T{\mathbf{E}}_{\mathcal{F}}$$

$$\hat{\mathbf{J}}=\left[\begin{array}{cccc}{\hat{\mathbf{J}}}_{{\bar{S}}_1}& & & \\ & {\hat{\mathbf{J}}}_{{\bar{S}}_2}& & \\ & & \ddots & \\ & & & {\hat{\mathbf{J}}}_{{\bar{S}}_R}\end{array}\right]$$

$$({\mathbf{E}}_{\mathcal{F}}^T{\mathbf{E}}_{\mathcal{F}}{)}^{-1}={\hat{\mathbf{J}}}^{-1}-\frac{1}{1+\mathrm{tr}\left(\mu {\mathbf{J}}_{\bar{S}}{\hat{\mathbf{J}}}^{-1}\right)}{\hat{\mathbf{J}}}^{-1}\mu {\mathbf{J}}_{\bar{S}}{\hat{\mathbf{J}}}^{-1}.$$

$${\hat{\mathbf{J}}}^{-1}=\left[\begin{array}{cccc}{\hat{\mathbf{J}}}_{{\bar{S}}_1}^{-1}& & & \\ & {\hat{\mathbf{J}}}_{{\bar{S}}_2}^{-1}& & \\ & & \ddots & \\ & & & {\hat{\mathbf{J}}}_{{\bar{S}}_R}^{-1}\end{array}\right],$$

$${\hat{\mathbf{J}}}_{{\bar{S}}_i}^{-1}$$

$$\mu {\mathbf{J}}_{\bar{S}}{\hat{\mathbf{J}}}^{-1}=\left[\begin{array}{ccc}\left(\frac{\mu }{L-\mu {\bar{S}}_1}\right){\mathbf{J}}_{\bar{S}\times {\bar{S}}_1}& \cdots & \left(\frac{\mu }{L-\mu {\bar{S}}_R}\right){\mathbf{J}}_{\bar{S}\times {\bar{S}}_R}\end{array}\right],$$

$${\hat{\mathbf{J}}}^{-1}\mu {\mathbf{J}}_{\bar{S}}{\hat{\mathbf{J}}}^{-1}=\left[\begin{array}{cccc}{\mathbf{A}}_{1,1}& {\mathbf{A}}_{1,2}& \cdots & {\mathbf{A}}_{1,R}\\ {\mathbf{A}}_{2,1}& {\mathbf{A}}_{2,2}& \cdots & {\mathbf{A}}_{2,R}\\ {\mathbf{A}}_{R,1}& {\mathbf{A}}_{R,2}& \cdots & {\mathbf{A}}_{R,R}\end{array}\right],$$

$${\mathbf{A}}_{i,j}=\mu \left(\frac{1}{L-\mu {\bar{S}}_i}\right)\left(\frac{1}{L-\mu {\bar{S}}_j}\right){\mathbf{J}}_{{\bar{S}}_i\times {\bar{S}}_j}.$$

$${\mathrm{err}}_{\mathcal{F}}\left(\mathbf{E}\right)$$

$$h(\mathcal{G})=\underset{\varnothing \ne \mathcal{F}\subset \mathcal{V}}{\min }\frac{|\partial \mathcal{F}|}{\min \{|\mathcal{F}|,|\mathcal{V}\setminus \mathcal{F}|\}}.$$

$$\frac{L-{\lambda }_2}{2}\le h(\mathcal{G})\le \sqrt{2L(L-{\lambda }_2)}.$$

Full text

In this work, our goal is to construct approximate gradient codes that are resilient to stragglers selected by a computationally unbounded adversary. Our motivation for constructing codes to mitigate adversarial stragglers stems from the challenge of tackling stragglers in massive-scale elastic and serverless systems, wherein it is difficult to statistically model stragglers. Towards this end, we propose a class of approximate gradient codes based on balanced incomplete block designs (BIBDs). We show that the approximation error for these codes depends only on the number of stragglers, and thus, adversarial straggler selection has no advantage over random selection. In addition, the proposed codes admit computationally efficient decoding at the server. Next, to characterize fundamental limits of adversarial straggling, we consider the notion of adversarial threshold – the smallest number of workers that an adversary must straggle to inflict certain approximation error. We compute a lower bound on the adversarial threshold, and show that codes based on symmetric BIBDs maximize this lower bound among a wide class of codes, making them excellent candidates for mitigating adversarial stragglers.

††This work is supported in part by National Science Foundation grants CCF-1748585 and CNS-1748692.

I Introduction

In many real-world applications, the size of training datasets has grown significantly over the years to the point that it is becoming crucial to implement learning algorithms in a distributed fashion. However, in practice the gains due to parallelization are often limited due to stragglers – workers that are slowed down due to unpredictable factors such as network latency, hardware failures, etc. [1, 2]. For instance, recent studies [3, 4] have demonstrated that straggling machines may run $\times 5$ to $\times 8$ slower than a typical machine on Amazon EC2. The straggler problem is even more daunting in massive-scale computing systems such as [5], which use AWS Lambda. Left untreated, stragglers severely impact latency, as the performance in each iteration is determined by the slowest machine.

Conventional approaches to mitigate stragglers involve detecting stragglers, ignoring stragglers, or replicating jobs across workers (see, e.g., [6, 7, 8, 9]). Recently, using coding-theoretic ideas to mitigate stragglers has gained significant research attention, see, e.g., [10, 11, 12, 13] for distributed computing, and [3, 14, 15, 16, 17] for distributed learning.

A coding theoretic framework for mitigating stragglers in distributed gradient-based learning methods was first proposed in [3]. The setup consists of $N$ worker machines and a parameter server. Training examples are partitioned into $K$ parts, and every worker is assigned $L$ of the $K$ parts. Each worker computes the partial gradient on its assigned examples, linearly combines the results according to some pre-specified vector of coefficients, and returns the result to the server. Note that the parameter $L$ essentially specifies the computation and storage load on individual workers. The authors showed that by redundantly assigning the parts across the workers and by judiciously choosing the coefficients of the linear combination at each worker, it is possible to exactly recover the sum of all gradients even if any $S$ workers straggle, and fail to return their results. Alternate code constructions for gradient coding have been proposed in [11, 14, 15, 18].

Gradient coding schemes designed for exactly recovering the gradient sum have two limitations. First, they fundamentally require heavy computational and storage overhead at each worker. In particular, in [3], it was established that any coding scheme designed to tolerate $S$ stragglers must have $L\geq K(S+1)/N$. This implies that the higher the straggler tolerance required, the larger is the computation and storage overhead per worker. Second, since the schemes are designed for a particular number of stragglers $S$, it is necessary to have an estimate on $S$ at the design time. This is not feasible for many practical schemes as straggler behavior can vary unpredictably.

In [15], the authors showed that these limitations can be lifted by allowing the server to approximately recover the gradient sum. Indeed, in many practical learning algorithms, it is sufficient to approximately reconstruct the gradient sum. The authors construct codes based on expander graphs, for which, the $\ell_{2}$-error of the approximate gradient sum, referred to as approximation error, degrades gracefully with the number of stragglers. These so-called approximate gradient codes do not require to have an estimate of the number of stragglers $S$ a priori, and allow the computation and storage overhead per worker to be substantially small.

In [19], the authors evaluate three families of approximate gradient codes: fractional repetition codes (FRCs), Bernoulli gradient codes (BCGs), and regularized BCGs based on sparse random graphs. They show that FRCs achieve small approximation error when the stragglers are chosen at random. However, FRCs perform poorly for adversarial stragglers, wherein an adversary can force to straggle any subset of workers up to a given size. Further, it is shown that adversarial straggler selection in general codes is NP-hard. In [20], the authors propose stochastic block codes (SBCs), which make it difficult for a computationally limited (polynomial-time bounded) adversary to select stragglers.

In this work, our goal is construct approximate gradient codes that can mitigate adversarial stragglers even for a computationally unbounded adversary. Our key idea is to construct codes based on combinatorial block designs. A block design is a family of subsets of a (finite) set, chosen in such a way that certain symmetry properties are satisfied (see [21] for details). We note that codes resilient to adversarial stragglers are useful in massive-scale elastic and serverless systems (such as [5]), wherein it is difficult to statistically model stragglers. Furthermore, we are interested in understanding fundamental limits of adversarial straggler selection.

Our Contributions: We propose a class of gradient codes based on balanced incomplete block designs (BIBDs) for mitigating adversarial stragglers. We show that the approximation error for these codes depends only on the number of stragglers, and not on which specific set of workers is straggling. Therefore, an adversary that can intelligently select stragglers has no advantage over one that chooses an arbitrary subset of stragglers. Moreover, for the proposed codes, the decoding vector at the server can be computed in closed-form. This enables the server to perform the decoding in a computationally efficient manner.

Next, we define the notion of adversarial threshold for a gradient code. The adversarial threshold specifies the minimum number of workers that an adversary must straggle to enforce that the approximation error is above a given target. We compute a lower bound on the adversarial threshold. Further, we show that codes based on symmetric BIBDs are excellent candidates for mitigating adversarial stragglers, since they maximize this lower bound among a wide class of codes.

II Framework

Notation: We use standard script for scalars, bold script for vectors and matrices, and calligraphic letters for sets. For a positive integer $n$, let $[n]=\{1,2,\ldots,n\}$. For a vector $\boldsymbol{v}$, let $\mathrm{supp}\left(\boldsymbol{v}\right)$ denote the support of $\boldsymbol{v}$. For a matrix $\boldsymbol{H}$, let $\boldsymbol{H}^{T}$ be its transpose, $\boldsymbol{H}^{\dagger}$ be its Moore-Penrose inverse, $\boldsymbol{H}_{i,j}$ be its $(i,j)$-th entry, $\boldsymbol{H}_{j}$ be its $j$-th column, and $\boldsymbol{H}_{\mathcal{T}}$ be the sub-matrix of $\boldsymbol{H}$ corresponding to the columns indexed by a set $\mathcal{T}\subset[n]$. Let $\boldsymbol{1}_{m}$ denote the $m\times 1$ all ones vector, and $\boldsymbol{I}_{m}$ denote the $m\times m$ identity matrix. Let $\boldsymbol{J}_{m\times n}$ and $\boldsymbol{0}_{m\times n}$ denote the $m\times n$ all ones and all zero matrices, respectively; when $m=n$, we simplify the notation to $\boldsymbol{J}_{m}$ and $\boldsymbol{0}_{m}$.

II-A Distributed Training

The process of learning the parameters $\mathbf{w}\in\mathbb{R}^{d}$ of a model given a dataset $D=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{M}$ of $M$ samples, where $\mathbf{x}_{i}\in\mathbb{R}^{d}$ and $y_{i}\in\mathbb{R}$, can be cast as the empirical risk minimization (ERM) problem given as

[TABLE]

where $\ell\left(\mathbf{x}_{i},y_{i};\mathbf{w}\right)$ is a loss function that measures the accuracy of the prediction made by the model $\mathbf{w}$ on the sample $(\mathbf{x}_{i},y_{i})$.

One popular method to approximately solve the ERM is stochastic gradient descent (SGD). SGD begins with some initial guess of $\mathbf{w}$ as $\mathbf{w}^{(0)}$, and then iteratively updates the parameters as $\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\alpha_{t}\nabla\ell\left(\mathbf{x}_{i_{t}},y_{i_{t}};\mathbf{w}^{(t)}\right)$, where $i_{t}$ is a sample index chosen randomly from $[M]$, and $\alpha_{t}>0$ is the learning rate (or step size) at iteration $t$.

In a distributed setting, it is possible to take advantage of parallelism by using mini-batch SGD. In every iteration of mini-batch SGD, a (possibly random) subset $S_{t}$ of $B$ samples is chosen and the model is updated as $\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\alpha_{t}}{B}\sum_{i\in S_{t}}\nabla\ell\left(\mathbf{x}_{i},y_{i};\mathbf{w}^{(t)}\right)$.

Next, we describe the framework of gradient coding which mitigates stragglers in a distributed implementation of mini-batch SGD by redundantly assigning gradients to workers.

Remark 1

Even though we focus on mini-batch SGD for the ease of exposition, our proposed coding techniques can be applied to other common first-order methods in machine learning. Moreover, our techniques are applicable to any distributed algorithm that requires the sum of multiple functions.

II-B Gradient Coding

Consider a distributed master-worker setting consisting of $N$ worker machines $W_{1}$, $W_{2}$, $\ldots$, $W_{N}$, and a parameter server. We focus our attention to a given iteration $t$, and fix a batch of $B$ samples $S_{t}$. Without loss of generality, assume that $S_{t}=\{1,2,\ldots,B\}$. We omit the explicit dependence on the iteration $t$ hereafter, since our focus is on a given iteration.

We partition the batch into $K$ subsets of equal size111We assume $K\mid B$ for simplicity. Our schemes can be easily adapted when $K\nmid B$., denoted as $D_{1},D_{2},\ldots,D_{K}$. Define the gradient vector of the partial data $D_{i}$, called partial gradient, as $\boldsymbol{g}_{i}:=\sum_{\mathbf{x}_{j},\mathbf{y}_{j}\in D_{i}}\nabla\ell\left(\mathbf{x}_{j},y_{j};\mathbf{w}\right)$. Note that the server is interested in computing $\boldsymbol{g}:=\sum_{i=1}^{K}\boldsymbol{g}_{i}$.

A gradient code (GC) consists of an encoding matrix $\boldsymbol{E}\in\mathbb{R}^{K\times N}$. The $j$-th column $\boldsymbol{E}_{j}$ of $\boldsymbol{E}$ corresponds to worker $j$, and determines which samples are assigned to the worker and what linear combination of gradients it returns to the server. In particular, let $\mathrm{supp}\left(\boldsymbol{E}_{j}\right)=L_{j}$. Then, the $j$-th worker is assigned the subsets $\{D_{i}:i\in L_{j}\}$, and it sends back to the server $\boldsymbol{c}_{j}=\frac{1}{K}\sum_{i\in L_{j}}\boldsymbol{g}_{i}\boldsymbol{E}_{i,j}$.

Let $L=\max_{j\in[N]}\left|\mathrm{supp}\left(\boldsymbol{E}_{j}\right)\right|$ and $R=\min_{j\in[K]}\left|\mathrm{supp}\left((\boldsymbol{E}^{T})_{j}\right)\right|$. We refer to $L$ as the computation load of $\boldsymbol{E}$ since a worker works on at most $L$ gradients. Note that for load balancing, it is good to assign the same number of gradient computations to each worker. We refer to $R$ as the replication factor of $\boldsymbol{E}$ since each gradient is computed by at least $R$ workers. We denote such a gradient code as an $(N,K,L,R)$-GC. (We summarize the notation in Table I.)

Decoding consists of finding a linear combination of the results from the non-straggling workers to approximate the gradient sum $\boldsymbol{g}$. Specifically, given a set of non-stragglers $\mathcal{F}\subset[N]$ of size $|\mathcal{F}|=N-S$, the server finds a vector $\boldsymbol{v}\in\mathbb{R}^{N-S}$, and computes $\hat{\boldsymbol{g}}=\boldsymbol{C}_{\mathcal{F}}\boldsymbol{v}$, where $\boldsymbol{C}=\begin{bmatrix}\boldsymbol{c}_{1}&\boldsymbol{c}_{2}&\cdots&\boldsymbol{c}_{N}\end{bmatrix}$.

Next, we use the framework of [19] (see also [15, 20]) to define the approximation error and the optimal decoding vector for a given gradient code as follows.

Definition 1

Given an encoding matrix $\boldsymbol{E}$, the approximation error $\mathrm{err}_{\mathcal{F}}\left(\boldsymbol{E}\right)$ for a given set of non-stragglers $\mathcal{F}\subseteq[N]$ of size $N-S$ is defined as

[TABLE]

and a solution $\boldsymbol{v}_{\mathrm{opt}}$ to (2) is called an optimal decoding vector. The worst-case approximation error for $S$ $(<N)$ stragglers is defined as

[TABLE]

Note that the deviation of $\hat{\boldsymbol{g}}$ from $\boldsymbol{g}$ can be bounded in terms of $\mathrm{err}_{\mathcal{F}}\left(\boldsymbol{E}\right)$ as as $\left\lVert\hat{\boldsymbol{g}}-\boldsymbol{g}\right\rVert_{2}^{2}\leq\left\lVert\boldsymbol{G}\right\rVert_{2}^{2}\mathrm{err}_{\mathcal{F}}\left(\boldsymbol{E}\right)$, where $\boldsymbol{G}$ is the matrix consisting of all the gradient vectors [15]. Our goal is to construct encoding matrices such that the worst-case approximation error is small. In addition, it is desirable if an optimal decoding vector can be computed efficiently.

III Preliminaries on Block Designs

We briefly review the relevant notions from the theory of block designs. For details, we refer the reader to [21].

Definition 2

[Design and Incidence Matrix] A design is a pair $(X,\mathcal{A})$, where $X$ is a set of elements called points, and $\mathcal{A}$ is a collection of nonempty subsets of $X$ called blocks. Consider a design $(X,\mathcal{A})$ with $X=\{x_{1},x_{2},\ldots,x_{v}\}$ and $\mathcal{A}=\{A_{1},A_{2},\ldots,A_{b}\}$. Then, the incidence matrix of $(X,\mathcal{A})$ is a $v\times b$ binary matrix $\boldsymbol{M}$ such that $\boldsymbol{M}_{i,j}=1$ if $x_{i}\in A_{j}$ and $\boldsymbol{M}_{i,j}=0$ if $x_{i}\notin A_{j}$.

Balanced incomplete block designs are probably the most-studied type of designs. They are defined as follows.

Definition 3

[BIBD] A $(v,b,k,r,\lambda)$-balanced incomplete block design (BIBD) is a design $(X,\mathcal{A})$ with $v$ points and $b$ blocks, each of size $k$, such that every point is contained in exactly $r$ blocks and any pair of distinct points is contained in exactly $\lambda$ blocks. (We summarize the notation in Table II.)

Remark 2

Note that the incidence matrix of a $(v,b,k,r,\lambda)$-BIBD is such that its every column contains exactly $k$ ones, every row contains exactly $r$ ones, and any two distinct rows intersect in exactly $\lambda$ locations. It is well-known that the parameters $v$, $b$, $k$, $r$, and $\lambda$ of a $(v,b,k,r,\lambda)$-BIBD should be such that $vr=bk$ and $r(k-1)=\lambda(v-1)$.

Example 1

[Fano Plane] A (symmetric) $(7,7,3,3,1)$-BIBD: $X=\{1,2,\ldots,7\}$ and $\mathcal{A}=\{123,145,167,246,257,347,356\}$. (To save space, we write blocks in the form $abc$ rather than $\{a,b,c\}$.) Observe that every block contains 3 points, and every point occurs in 3 blocks. In addition, every pair of distinct points is contained in exactly one block.

IV Gradient Codes Using BIBDs

In this section, we consider gradient codes based on BIBDs. For any $(v,b,k,r,\lambda)$-BIBD $(X=\{X_{1},\ldots,X_{v}\},\mathcal{A}=\{\mathcal{A}_{1},\ldots,\mathcal{A}_{b}\})$, let us construct a gradient code using the BIBD in the following natural way. Consider a distributed system with $N=b$ workers. Partition the training dataset into $K=v$ subsets $D_{1},D_{2},\ldots,D_{v}$, and allocate a subset $D_{i}$ to worker $j$ if the $i$-th point belongs to the $j$-th block, i.e., if $X_{i}\in\mathcal{A}_{j}$. By the definition of a BIBD, each worker will compute $L=k$ gradients and each gradient will be computed $R=r$ times. This construction can be concisely described in terms of the incidence matrix as follows.

Construction 1

Given a $(v,b,k,r,\lambda)$-BIBD with incidence matrix $\boldsymbol{M}$, construct a gradient code with the encoding matrix $\boldsymbol{E}=\boldsymbol{M}$. The resulting gradient code is an $(N=b,K=v,L=k,R=r)$-GC.

We note that the parameters of any code constructed using a BIBD are restricted to $NL=KR$ and $R(L-1)=\lambda(K-1)$ (see Remark 2). On the other hand, since BIBDs have received significant research attention in combinatorics and a large number of constructions have been proposed (see e.g. [22]), this enables us to construct a class of gradient codes for a wide range of parameters.

In the following sections, we focus our attention to codes constructed using three well-studied families of BIBDs. We show that these codes have two key advantages. First, combinatorial properties of BIBDs enable us to compute an optimal decoding vector in closed-form. This results in extremely simple and efficient decoding at the server. Secondly, these codes are resilient to adversarial stragglers.

IV-A Gradient Codes Using Symmetric BIBDs

Symmetric designs form an important class of block designs. The well-known Fisher’s inequality for BIBDs states that, for any $(v,b,k,r,\lambda)$-BIBD, the parameters $v$ and $b$ should satisfy $b\geq v$. A $(v,b,k,r,\lambda)$-BIBD in which $v=b$ (or, equivalently $r=k$) is called a symmetric $(v,b,k,r,\lambda)$-BIBD.222It is worth noting that the incidence matrix of a symmetric BIBD need not be a symmetric matrix. Using symmetric BIBDs in Construction 1 results in a class of gradient codes with $N=K=v$,333We note that the parameter regime $N=K$ has received primary research attention, see e.g. [3, 15, 19, 20, 16]. and $L=R=k$.

Remark 3

It is well-known that any pair of distinct blocks of a symmetric $(v,b,k,r,\lambda)$-BIBD intersect in exactly $\lambda$ points. This ensures that for a gradient code constructed using a symmetric BIBD, any pair workers share exactly $\lambda$ gradients.

This property enables us to characterize the approximation error as well as optimal decoding vector in closed form.

Theorem 1

Consider an $(N,K,L,R)$-GC obtained from a symmetric BIBD using Construction 1.

For any set of non-stragglers $\mathcal{F}$ of size $(N-S)$, an optimal decoding vector is

[TABLE] 2. 2.

The worst-case approximation error for $S$ stragglers is

[TABLE]

Proof:

(Sketch) The key idea is to show that for any set of non-stragglers of given size, it is possible to solve the normal equation. This relies on the property that any pair of workers share exactly $\lambda$ gradient computations. The main technical tool that we use is a matrix inversion lemma from [23] for the inverse of the sum of two matrices (see Appendix A). The complete proof is deferred to Appendix B. ∎

Remark 4

Note that the optimal decoding vector (4) and worst-case decoding error (5) depend only on the number of stragglers and not on the specific set of stragglers. Therefore, decoding at the server can be performed in a very efficient way. Moreover, since any set of $S$ stragglers is as harmful as other (in terms of the approximation error), an adversary cannot do better than straggling an arbitrary set of $S$ stragglers. This makes these codes resilient to adversarial straggling.

Note that it is possible to construct several classes of gradient codes based on well-known families of symmetric BIBDs. We present a few examples in the following.

IV-A1 Class of Gradient Codes Based on Projective Geometries

For any prime power $q$ and integer $m\geq 2$, the projective geometry of order $q$ and dimension $m$ can be used to obtain a symmetric BIBD [21]. Using such a BIBD in Construction 1 yields a class of gradient codes with the following parameters: $N=K=(q^{m+1}-1)/(q-1)$, $L=K=(q^{m}-1)/(q-1)$. Any pair of distinct workers share exactly $\lambda=(q^{m-1}-1)/(q-1)$ gradient computations.

As we will see in Sec. V, gradient codes based on projective planes (i.e., $m=2$) are nearly optimal in terms of the worst-case approximation error when $S=O(q)$ and $q$ is sufficiently large.

IV-A2 Class of Gradient Codes Based on Hadamard Designs

For a positive integer $m\geq 2$, a symmetric BIBD can be constructed using a Hadamard matrix of order $4m$ [21]. Using such a BIBD in Construction 1 yields a class of gradient codes with the following parameters: $N=K=4m-1$, $L=K=2m-1$. Any pair of distinct workers share exactly $\lambda=m-1$ gradient computations.

IV-B Gradient Codes Using Dual Designs

Given a $(v,b,k,r,\lambda)$-BIBD $(X,\mathcal{A})$ with the incidence matrix $\boldsymbol{M}$, the design having incidence matrix $\boldsymbol{M}^{T}$ is called the dual design of $(X,\mathcal{A})$. When the dual of a $(v,b,k,r,\lambda)$-BIBD is used in Construction 1, it is easy to see that the resulting code is a $(N=v,K=b,L=r,R=k)$-GC. Note that, unlike symmetric BIBDs, using dual designs allows us to construct codes for which $N\neq K$.

Since every pair of distinct points is contained in $\lambda$ number of blocks in a BIBD, any two distinct blocks of the dual intersect in exactly $\lambda$ points.

Theorem 2

Consider an $(N,K,L,R)$-GC with encoding matrix $\boldsymbol{E}$ obtained from the dual of a BIBD using Construction 1. Then, for any set of non-stragglers $\mathcal{F}$ of size $(N-S)$, an optimal decoding vector is given by (4) and the worst-case approximation error for $S$ stragglers is given by (5).

Proof:

The proof of Theorem 1 relies on the property that any two blocks of a symmetric BIBD intersect in exactly $\lambda$ points. Since the same property holds for the dual of a BIBD, the proof is identical to that of Theorem 1. ∎

Note that codes constructed from duals of BIBDs also admit computationally efficient decoding and are resilient to adversarial straggling by the same arguments as in Remark 4.

It is possible to construct several classes of gradient codes by considering duals of well-known families of BIBDs. We present a few examples in the following.

IV-B1 Class of Gradient Codes Based on the Duals of Affine Geometries

For any power of prime $q$ and integer $m\geq 2$, the affine geometry of order $q$ and dimension $m$ can be used to obtain a BIBD [21]. Using the dual of such a BIBD in Construction 1 yields a class of gradient codes with the following parameters: $N=q^{m}$, $K=q(q^{m}-1)/(q-1)$, $L=(q^{m}-1)/(q-1)$, $R=q^{m-1}$ such that any pair of distinct workers share $\lambda=(q^{m-1}-1)/(q-1)$ gradient computations.

IV-B2 Class of Gradient Codes Based on the Duals of Residual and Derived Designs

Derived and residual BIBDs are well-known methods to obtain new BIBDs from symmetric BIBDs (see [21, Chapter 2.2]). Using the duals of these designs allows us to construct gradient codes for a broad class of parameters.

IV-C Gradient Codes Using Resolvable Designs

Consider a gradient code with replication factor $R$. If the number of stragglers $S<R$, then every gradient is computed by at least one of the remaining workers. Note that any exact gradient code can recover the gradient sum in this case. Therefore, it is desirable to construct approximate gradient codes that can exactly recover the gradient sum whenever $S<R$. However, as we can see from (5) that this is not the case for gradient codes obtained using either symmetric BIBDs or dual designs.

In this section, we consider gradient codes based on a special class of block designs called resolvable designs that lift this limitation. We begin with the definition of a resolvable BIBD.

Definition 4

[Resolvable BIBD] A parallel class in a design is a subset of disjoint blocks whose union is the point set. Let $(X,\mathcal{A})$ be a $(v,b,k,r,\lambda)$-BIBD. A partition of $\mathcal{A}$ into $r$ parallel classes is called a resolution. A $(v,b,k,r,\lambda)$-BIBD is said to be a resolvable BIBD if $\mathcal{A}$ has at least one resolution.

Remark 5

Note that a parallel class contains exactly $v/k$ blocks. Further, for any resolvable BIBD, it must be that $b\geq v+r-1$, or, equivalently, $r\geq k+\lambda$ (known as Bose’s inequality).

In the rest of this section, we focus our attention to a well-studied class of resolvable BIBDs called affine resolvable BIBDs. A resolvable $(v,b,k,r,\lambda)$-BIBD with $b=v+r-1$ (or, equivalently, $r=k+\lambda$) is said to be an affine resolvable BIBID.

Example 2

[Affine Plane of Order 2] A resolvable $(9,12,3,4,1)$-BIBD $X=\{1,2,\ldots,9\}$ and $\mathcal{A}=\{\mathcal{P}_{1},\mathcal{P}_{2},\mathcal{P}_{3},\mathcal{P}_{4}\}$, where $\mathcal{P}_{1}=\{123,456,789\}$, $\mathcal{P}_{2}=\{147,258,369\}$, $\mathcal{P}_{3}=\{159,267,348\}$, and $\mathcal{P}_{4}=\{168,249,357\}$. Note that each $\mathcal{P}_{i}$ is a parallel class and the partition $\{\mathcal{P}_{1},\ldots,\mathcal{P}_{4}\}$ forms a resolution.

Consider a gradient code obtained from a resolvable $(v,b,k,r,\lambda)$-BIBD using Construction 1. The resulting code is an $(N=b,K=v,L=k,R=r)$-GC.

Remark 6

Given an arbitrary resolution of the blocks $\mathcal{A}$ as $\{\mathcal{P}_{1},\mathcal{P}_{2},\ldots,\mathcal{P}_{r}\}$, the $N$ workers can be partitioned into $R$ $(=r)$ sets $\{\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{R}\}$ such that the $j$-th worker is included in set $\mathcal{T}_{i}$ if the $j$-th block is in the set $\mathcal{P}_{i}$. Then, naturally, for every $i\in[R]$, any pair of distinct workers in $\mathcal{T}_{i}$ compute disjoint gradients. Moreover, for every part $\mathcal{T}_{i}$, $i\in[R]$, workers in $\mathcal{T}_{i}$ together compute all the gradients. Therefore, if the server receives the results from all the workers in any $\mathcal{T}_{i}$, it can exactly recover the gradient sum by simply adding the partial gradient sums.

Remark 7

*Consider an affine resolvable $(v,b,k,r,\lambda)$-BIBD $(X,\mathcal{A})$ with incidence matrix $\boldsymbol{M}$. It is well-known that any two blocks from different parallel classes of an affine resolvable BIBD intersect in exactly $k^{2}/v$ points.444The parameters of an affine BIBD are such that $k^{2}/v$ is an integer. Therefore, any worker from $\mathcal{T}_{i}$ shares exactly $L^{2}/K$ gradients with any worker from $\mathcal{T}_{j}$ such that $j\neq i$. For simplicity, define $\mu:=L^{2}/K$. *

The above property enables us to characterize the approximation error as well as optimal decoding vector in closed form. However, the analysis in this case turns out to be more intricate than the case of symmetric (or dual) BIBDs.555The increased complexity of analysis can be attributed to the fact that any pair of workers for an affine design share either zero gradients or $L^{2}/K$ gradients. On the other hand, for a symmetric (or dual) BIBD, any pair of workers share the same number of gradients. In fact, affine resolvable designs belong to a class of designs called quasi-symmetric designs: designs with the property that any pair of blocks intersect in either $x$ or $y$ points. Towards this, we need to introduce the following notation.

Definition 5

Consider a set of non-stragglers $\mathcal{F}$ of size $(N-S)$. Define $\mathcal{F}_{i}:=\mathcal{F}\cap\mathcal{T}_{i}$ and $S_{i}:=N/L-|\mathcal{F}_{i}|$. Note that $S_{i}$ denotes the number of stragglers among the workers from $\mathcal{T}_{i}$, and that $0\leq S_{i}\leq N/L$ and $\sum_{i=1}^{R}S_{i}=S$. We call $[S_{1}\>\>S_{2}\>\>\cdots\>\>S_{R}]$ as the straggler profile corresponding to the set $\mathcal{F}$.

Theorem 3

Consider an $(N,K,L,R)$-GC with encoding matrix $\boldsymbol{E}$ obtained from an affine resolvable BIBD using Construction 1. Consider a set of non-stragglers $\mathcal{F}$ of size $(N-S)$ with straggler profile $[S_{1}\>\>S_{2}\>\>\cdots\>\>S_{R}]$. Define $\hat{S}_{0}:=0$ and $\hat{S}_{i}:=\sum_{p=1}^{i}\left(K/L-S_{p}\right)$ for $i\in[R]$. Recall that $\mu:=L^{2}/K$.

If there exists an $i\in[R]$ such that $S_{i}=0$, then an optimal decoding vector is

[TABLE]

and the corresponding approximation error is $\mathrm{err}_{\mathcal{F}}\left(\boldsymbol{E}\right)=0$. 2. 2.

Suppose $S_{i}>0$ for all $i\in[R]$, then an optimal decoding vector is

[TABLE]

for $\hat{S}_{i-1}+1\leq j\leq\hat{S}_{i}$, and the corresponding approximation error is

[TABLE]

where $c_{i}=\frac{L/\left(L-\mu(K/L-S_{i})\right)}{\left(1+\sum_{p=1}^{R}\frac{\mu(K/L-S_{p})}{L-\mu(K/L-S_{p})}\right)}$.

Proof:

(Sketch) For the first part, note that all the workers from $\mathcal{T}_{i}$ return their computations. From Remark 6, simply summing the results from the workers in $\mathcal{T}_{i}$ recovers the gradient sum. The vector in (6) computes this sum. Clearly, the approximation error in this case is zero.

For the second part, we show that it is possible to solve the normal equation by computing $\boldsymbol{E}^{\dagger}$ in closed formed. For this, we leverage the block intersection property of affine resolvable designs mentioned in Remark 7. In this case, we need to iteratively use the matrix inversion lemma given in Appendix A. The complete proof is deferred to Appendix C. ∎

Remark 8

Note that since optimal decoding vector depends only on the straggler profile (number of stragglers from each set $\mathcal{T}_{i}$, $i\in[R]$), decoding at the server can be performed in a very efficient way. Further, any adversary that can enforce at most $S_{i}$ stragglers from set $\mathcal{T}_{i}$, $i\in[R]$, cannot worsen the error by intelligently selecting stragglers as opposed to randomly selecting stragglers.

IV-D Summary of Constructions

It is possible to construct several classes of gradient codes based on well-known families of BIBDs. We summarize a few examples in Table III.

V Performance Evaluation

In this section, we numerically evaluate the performance of the proposed design-based schemes. We consider the following gradient coding schemes (see Table III, fix $m=2$): (i) $(N=q^{2}+q+1,K=q^{2}+q+1,L=q+1,R=q+1)$-GC based on the projective plane of order $q$ (denoted as PG), (ii) $(N=q^{2},K=q^{2}+q,L=q+1,R=q)$-GC based on the dual of affine plane of order $q$ (denoted as Dual AG), and (iii) $(N=q^{2}+1,N=q^{2},L=q,R=q+1)$-GC based on the affine plane of order $q$ (denoted as AG).

We plot the worst-case approximation error normalized by the number of gradients, i.e., $\mathrm{err}_{S}\left(\boldsymbol{E}\right)/K$ versus the normalized number of stragglers, i.e., $S/N$. Specifically, we consider the following two regimes: $q=5$ and $q=9$ in Figures 1(a) and 1(b), respectively. Observe that the three families PG, Dual AG, and AG have similar performance in terms of approximation error.

For comparison, we plot the uncoded case which partitions $K=q^{2}+q+1$ gradients across $N=q^{2}+q+1$ workers. Note that the approximation error in this case equals the number of stragglers. In addition, we also consider an $(N=(q+1)^{2},K=(q+1)^{2},L=q+1,R=q+1)$-fractional repetition code (FRC) [3, 19]. As expected, both the uncoded and FRC schemes perform poorly when the stragglers are adversarial.

In addition, we consider codes based on Margulis construction of Ramanujan graphs in [15, Example 19], denoted as RG. For these codes, we plot the upper bound on the worst-case approximation error derived in [15] as a proxy for the worst-case approximation error. This is because, to obtain the worst-case approximation error, one needs to consider all possible subsets of stragglers. This becomes computationally infeasible for large $N$ and $S$. We see that the worst-case approximation error for BIBD-based codes is substantially smaller than the guarantees given by the upper bound for the RG scheme.

To see how well the proposed codes perform, we consider a lower bound on the worst-case approximation error from [15]. In particular, in [15, Lemma 21], the authors showed that for any $(N,K,L,R)$-GC $\boldsymbol{E}$ with $K=N$, the worst-case approximation error can be lower bounded as $\mathrm{err}_{S}\left(\boldsymbol{E}\right)\geq\lfloor S/L\rfloor$. From Fig. 1(a) and 1(b), we can observe that the proposed schemes perform close to this lower bound for the small number of stragglers. It is worth noting that, in massive-scale serverless systems, which are our motivation to mitigate adversarial stragglers, only a small number of machines straggle substantially (see, e.g., [24, Fig. 1]).

In fact, gradient codes based on projective planes are nearly optimal for large $q$ and $S=O(q)$. To see this, consider $(N=q^{2}+q+1,K=q^{2}+q+1,L=q+1,R=q+1)$-GC based on the projective plane of order $q$. The worst-case approximation error in (5) reduces to the following expression.

[TABLE]

Observe that when $S=O(q)$ and $q$ is large, the error above is close to the lower bound $\lfloor S/(q+1)\rfloor$.

VI Robustness Against Adversarial Straggling

Our goal in this section is to investigate fundamental limits on the approximation error. As mentioned in the previous section, for any $(N,K,L,R)$-GC $\boldsymbol{E}$ with $K=N$, the worst-case approximation error can be lower bounded as $\mathrm{err}_{S}\left(\boldsymbol{E}\right)\geq\lfloor S/L\rfloor$ [15, Lemma 21]. In fact, the proof of [15, Lemma 21] is constructive and gives an $O(N^{2})$ time greedy algorithm to find a set of stragglers that will enforce $\mathrm{err}_{S}\left(\boldsymbol{E}\right)\geq\lfloor S/L\rfloor$. In other words, even an adversary with limited computing power can induce the error of at least $\lfloor S/L\rfloor$. However, in general, for a given gradient code and a number $S$, finding a set of $S$ stragglers that maximize the approximation error is shown to be NP-hard in [19].666The authors consider the case when the decoding vector is fixed a priori. In particular, it is assumed that the decoding vector is of the form $\boldsymbol{v}=\rho\boldsymbol{1}_{N-S}$ for a fixed constant $\rho$.

To analyze fundamental limits for a computationally unbounded adversary, we consider the following problem: given a gradient code and a target $\eta$, what is the minimum number of stragglers that an adversary must introduce to ensure that the approximation error is at least $\eta$?

Towards this, consider a bipartite graph $\mathcal{G}=(\mathcal{W},\mathcal{D},\mathcal{E})$ for a given $(N,K,L,R)$-GC with encoding matrix $\boldsymbol{E}$ as follows. The left $N$ vertices $\mathcal{W}$ correspond to the set of workers, while the right $K$ vertices $\mathcal{D}$ correspond to the set of gradients to be computed. There is an edge $\{i,j\}\in\mathcal{E}$ from a vertex $i\in\mathcal{W}$ to a vertex $j\in\mathcal{D}$ iff $\boldsymbol{E}_{i,j}\neq 0$. Note that the graph $\mathcal{G}$ specifies the placement scheme for the gradient code, i.e., how the data parts are assigned to the workers.

Consider a set $\mathcal{T}\subset{\mathcal{D}}$ and let $\mathcal{N}\left(\mathcal{T}\right)\subset\mathcal{W}$ denote the neighbors of $\mathcal{T}$ in $\mathcal{G}$. Now, suppose all the workers in $\mathcal{N}\left(\mathcal{T}\right)$ are straggling. Then, the gradients in $\mathcal{T}$ cannot contribute to the gradient sum. Therefore, the approximation error must be at least $|\mathcal{T}|$. Based on this observation, we introduce the notion of adversarial threshold by defining the following adversarial straggling problem.

Definition 6

[Adversarial Threshold] Given a graph $\mathcal{G}$ associated with an $(N,K,L,R)$-GC and a constant $0<\eta<K$, define

[TABLE]

We refer to refer to the above minimization problem as the adversarial straggling problem, and $S^{*}(\eta)$ as the adversarial threshold.

Note that, given $\mathcal{G}$, $S^{*}(\eta)$ is the smallest number of workers that must be selected by an adversarial straggler to enforce that the approximation error is at least $\eta$.777We do not consider an encoding matrix $\boldsymbol{E}$ explicitly in the formulation for simplicity.

Next, we derive a lower bound on $S^{*}(\eta)$. We restrict our attention on a class $\mathcal{C}$ of gradient codes for which $N=K$, and the associated bipartite graph $\mathcal{G}$ is regular and connected.

Proposition 1

For any gradient code from the class $\mathcal{C}$, and for any $\eta\leq N/4$, we have

[TABLE]

where $\lambda_{2}$ is the second largest eigenvalue of the graph $\mathcal{G}$ associated with the code.888For a brief review of eignevalues of a graph, see Appendix D.

Proof:

See Appendix E. ∎

Next, we show that codes obtained from symmetric BIBDs are excellent candidates to mitigate adversarial stragglers, since they achieve the maximum $S^{*}(\eta)$ among the codes from $\mathcal{C}$.

Proposition 2

Let $\eta\leq N/4$. Gradient codes obtained from symmetric BIBDs via Construction 1 achieve the maximum value of $S^{*}(\eta)$ among the codes in $\mathcal{C}$.

Proof:

See Appendix F. ∎

Appendix A Matrix Inversion Lemma

Lemma 1

(cf. [23]) Let $\boldsymbol{G}$ and $\boldsymbol{G}+\boldsymbol{H}$ be nonsingular matrices where $\boldsymbol{H}$ is a matrix of rank one. Then, $\mathrm{tr}\left(\boldsymbol{H}\boldsymbol{G}^{-1}\right)\neq-1$, and the inverse of $(\boldsymbol{G}+\boldsymbol{H})$ is

[TABLE]

Appendix B Proof of Theorem 1

Consider an arbitrary set of non-stragglers $\mathcal{F}\subset[N]$ of size $(N-S)$. Define $\bar{S}:=N-S$. Recall that we have

[TABLE]

One optimal solution to the above least squares problem is $\boldsymbol{v}_{\mathrm{opt}}=\boldsymbol{E}_{\mathcal{F}}^{\dagger}\boldsymbol{1}_{K}$.

Since $\boldsymbol{E}=\boldsymbol{M}$, each column of $\boldsymbol{E}$ contains exactly $L$ ones and any two columns of $\boldsymbol{E}$ intersect in exactly $\lambda$ locations (see Remark 2). Therefore, we have

[TABLE]

Note that the matrix on the right hand side of (14) above has an eigenvalue $L-\lambda$ with multiplicity $\bar{S}-1$ and an eigenvalue $(L-\lambda)+\lambda\bar{S}$ with multiplicity one. Thus, its determinant is $(L-\lambda)^{\bar{S}-1}((L-\lambda)+\lambda\bar{S})\neq 0$.999For any BIBD, $k>\lambda$. Therefore, we have $L>\lambda$. Note that the same proof also works for Theorem 2, where we use a dual of a BIBD. In this case, since $r>\lambda$, we again have $L>\lambda$. Therefore, $\boldsymbol{E}_{\mathcal{F}}^{T}\boldsymbol{E}_{\mathcal{F}}$ is nonsingular, and we have $\boldsymbol{E}_{\mathcal{F}}^{\dagger}=(\boldsymbol{E}_{\mathcal{F}}^{T}\boldsymbol{E}_{\mathcal{F}})^{-1}\boldsymbol{E}_{\mathcal{F}}^{T}$.

Next, we compute $(\boldsymbol{E}_{\mathcal{F}}^{T}\boldsymbol{E}_{\mathcal{F}})^{-1}$ as follows:

[TABLE]

where (a) follows from (14), and (b) follows from Lemma 1 in Appendix A.

Now, we can compute $\boldsymbol{v}_{\mathrm{opt}}$ as

[TABLE]

where (c) follows from $\boldsymbol{v}_{\mathrm{opt}}=\boldsymbol{E}_{\mathcal{F}}^{\dagger}\boldsymbol{1}_{K}$, and (d) follows from (13). Finally, (4) follows from (16) noting that $\bar{S}=N-S$.

Next, we compute $\mathrm{err}_{\mathcal{F}}\left(\boldsymbol{E}\right)$ for an arbitrary set of non-stragglers $\mathcal{F}$ of size $\bar{S}$.

[TABLE]

where (e) follows from $\mathrm{err}_{\mathcal{F}}\left(\boldsymbol{E}\right)=\left\lVert\boldsymbol{E}_{\mathcal{F}}\boldsymbol{v}_{\mathrm{opt}}-\boldsymbol{1}_{K}\right\rVert_{2}^{2}$, (f) follows from (13) and (14), and (g) follows after substituting $\boldsymbol{v}_{\mathrm{opt}}$ from (16).

Since $\mathrm{err}_{\mathcal{F}}\left(\boldsymbol{E}\right)$ does not depend on the specific set of stragglers, but only the size of it, we get  (5) from (17) substituting $\bar{S}=N-S$.

Appendix C Proof of Theorem 3

Consider an arbitrary set of non-stragglers $\mathcal{F}$ of size $(N-S)$ with straggler profile $[S_{1}\>\>S_{2}\>\>\cdots\>\>S_{R}]$. Recall that $0\leq S_{i}\leq N/L$ and $\sum_{i=1}^{R}S_{i}=S$. Define $\bar{S}_{i}:=K/L-S_{i}$ for $i\in[R]$ and $\bar{S}_{0}:=0$. We consider the second case when $S_{i}>0$ for every $i\in[R]$.

Recall that we need to solve

[TABLE]

One optimal solution to the above least squares problem is $\boldsymbol{v}_{\mathrm{opt}}=\boldsymbol{E}_{\mathcal{F}}^{\dagger}\boldsymbol{1}_{K}$.

By following the proof of Bose’s inequality for resolvable block designs, we have that any sub-matrix of $\boldsymbol{E}$ with an arbitrary column removed from each of $\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{R}$ has full column rank. Since we have $S_{i}>0$ for every $i\in[R]$, it follows that $\boldsymbol{E}_{\mathcal{F}}$ has full column rank. Therefore, $\boldsymbol{E}_{\mathcal{F}}^{T}\boldsymbol{E}_{\mathcal{F}}$ is nonsingular, and we have $\boldsymbol{E}_{\mathcal{F}}^{\dagger}=(\boldsymbol{E}_{\mathcal{F}}^{T}\boldsymbol{E}_{\mathcal{F}})^{-1}\boldsymbol{E}_{\mathcal{F}}^{T}$.

From Remark 7, we obtain that

[TABLE]

where $\boldsymbol{\hat{J}}$ is a block matrix defined as

[TABLE]

such that $\boldsymbol{\hat{J}}_{\bar{S}_{i}}=L\boldsymbol{I}_{\bar{S}_{i}}-\mu\boldsymbol{J}_{\bar{S}_{i}}$ for $i\in[R]$. Note that we suppress the zero entries in the right hand side of (20) for simplicity.

Next, from (19) and Lemma 1 (in Appendix A), we get:

[TABLE]

Due to the block structure of $\boldsymbol{\hat{J}}$, we have

[TABLE]

where $\boldsymbol{\hat{J}}_{\bar{S}_{i}}^{-1}$ can be computed as follows:

[TABLE]

The equality (a) above is obtained using Lemma 1.

Using (22) and (23), we verify that

[TABLE]

and thus, $\textrm{tr}\left(\mu\boldsymbol{J}_{\bar{S}}\boldsymbol{\hat{J}}^{-1}\right)=\sum_{p=1}^{R}\frac{\mu\bar{S}_{p}}{L-\mu\bar{S}_{p}}$. Using (24), one can verify that

[TABLE]

where

[TABLE]

We verify, by substituting the above results in (21) and using (18), that $(\boldsymbol{E}_{\mathcal{F}}^{T}\boldsymbol{E}_{\mathcal{F}})^{-1}\boldsymbol{E}_{\mathcal{F}}^{T}\boldsymbol{1}_{K}$ results in the expression of $\boldsymbol{v}_{\mathrm{opt}}$ given in (7).

Finally, note that

[TABLE]

Expression (8) is obtained by using (18), (19), and (7) in (27).

Appendix D Overview of Expansion and Spectral Properties of a Graph

Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ be a finite, undirected and connected graph on $N$ vertices. For a subset of vertices $\mathcal{F}\subset\mathcal{V}$, the boundary $\partial\mathcal{F}$ is the set of edges connecting $\mathcal{F}$ to $\mathcal{V}\setminus\mathcal{F}$. The expanding constant or isopetimetric constant of $\mathcal{G}$ is defined as (see [25])

[TABLE]

It is well-known that expansion properties of a graph are closely related to the adjacency matrix $\boldsymbol{A}$ of the graph, defined as follows: $\boldsymbol{A}_{i,j}=1$ iff vertices $i$ and $j$ are connected by an edge, i.e., $\{i,j\}\in\mathcal{E}$. Since $\boldsymbol{A}$ is a real symmetric matrix, it has real eigenvalues $\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{N}$. When $\mathcal{G}$ is $L$-regular, it is well-known that $\lambda_{1}=L$ , $\lambda_{2}<L$ and $\lambda_{N}\geq-L$, where equality holds iff $\mathcal{G}$ is bipartite [25].

Theorem 4

(cf. [26]) If $\mathcal{G}$ is a finite, connected, $L$-regular graph, then

[TABLE]

Connected regular graphs for which $\lambda_{2}$ is smaller than the vertex degree are called as expanders.

Theorem 5

(cf. [27]) Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ be a connected, $(L,R)$-regular graph. Then

[TABLE]

For the $r$-regular graph of a symmetric $(v,b,k,r,\lambda)$-BIBD, the bound in (30) is satisfied with equality.

Appendix E Proof of Proposition 1

Let $\mathcal{G}=(\mathcal{W},\mathcal{D},\mathcal{E})$ be the associated bipartite graph. Consider any $\mathcal{T}\subset\mathcal{D}$ of size $\eta$ and let $\mathcal{S}=\mathcal{N}\left(\mathcal{T}\right)\subset\mathcal{W}$. Let $S=|\mathcal{S}|$. Note that there are $\eta L$ edges from $\mathcal{T}$ to $\mathcal{S}$. Further, there are $SL$ edges such that, for each edge, one of the endpoints is incident on $\mathcal{S}$. Let $\partial(\mathcal{S}\cup\mathcal{T})$ be the boundary of $\mathcal{S}\cup\mathcal{T}$. Recall that this is the set of edges connecting $\mathcal{S}\cup\mathcal{T}$ to $\{\mathcal{W}\cup\mathcal{D}\}\setminus\{\mathcal{S}\cup\mathcal{T}\}$. Therefore, we have $LS=L\eta+|\partial(\mathcal{S}\cup\mathcal{T})|$, from which, we get

[TABLE]

where (a) follows from (28) and $\eta\leq N/4$, and (b) follows from Theorem 4. The result follows after rearranging (34).

Appendix F Proof of Proposition 2

From (11), observe that the smaller the $\lambda_{2}$ the larger is $S^{*}_{LB}(\eta)$. Therefore, the result follows immediately from Theorem 5.