On the Average Locality of Locally Repairable Codes

TL;DR

This paper establishes bounds on the average locality of locally repairable codes (LRCs) and designs new codes that optimize this metric, improving repair efficiency in distributed storage systems.

Contribution

It provides the first lower bounds on the average locality of LRCs and introduces new code constructions that achieve these bounds for practical code rates.

Findings

Lower bound on average locality for arbitrary LRCs

Tight lower bound for high-rate LRCs

New LRC constructions achieving optimal average locality

Abstract

A linear block code with dimension $k$, length $n$, and minimum distance $d$ is called a locally repairable code (LRC) with locality $r$ if it can retrieve any coded symbol by at most $r$ other coded symbols. LRCs have been recently proposed and used in practice in distributed storage systems (DSSs) such as Windows Azure storage and Facebook HDFS-RAID. Theoretical bounds on the maximum locality of LRCs ($r$) have been established. The \textit{average} locality of an LRC ($\overline{r}$) directly affects the costly repair bandwidth, disk I/O, and number of nodes involved in the repair process of a missing data block. There is a gap in the literature studying $\overline{r}$. In this paper, we establish a lower bound on $\overline{r}$ of arbitrary $(n,k,d)$ LRCs. Furthermore, we obtain a tight lower bound on $\overline{r}$ for a practical case where the code rate $(R=\frac{k}{n})$ is greater than $(1-\frac{1}{\sqrt{n}})^2$. Finally, we design three classes of LRCs that achieve the obtained bounds on $\overline{r}$. Comparing with the existing LRCs, our proposed codes improve the average locality without sacrificing such crucial parameters as the code rate or minimum distance.

Equations

$$d\leq n-k-\Big{\lceil}\frac{k}{r}\Big{\rceil}+2,$$

$$\forall \text{leavevmode}\mathbf{x},\text{leavevmode}\text{leavevmode}\text{leavevmode}\sum _{l\in \Theta }{\alpha }_l{y}_l=0,\text{leavevmode}\text{leavevmode}\text{leavevmode}{\alpha }_l\in {\mathbb{F}}_q,{\alpha }_l\ne 0,$$

$$r+1\ge \frac{\gamma +k}{\gamma }\ge \frac{J+k}{J},m\in [J,n-k].$$

$$r+1\ge \frac{n}{m}\ge \frac{n}{J-1}>\frac{J+k}{J},m\in [2,J-1],$$

$$r\ge \frac{k}{J}\Rightarrow n-k-d+2\ge \frac{k}{r}\Rightarrow d\le n-k-\frac{k}{r}+2$$

$$\sum_{j=1}^{\zeta}z_{j}^{2}\geq(\zeta-a)\Big{\lfloor}\frac{A}{\zeta}\Big{\rfloor}^{2}+a\Big{\lceil}\frac{A}{\zeta}\Big{\rceil}^{2},$$

$$z_l>z_m+1\iff z_l^2+z_m^2\ge (z_l-1{)}^2+(z_m+1{)}^2.$$

$$d\leq n-k-\Big{\lceil}\frac{k}{r}\Big{\rceil}+2\leq n-k-\frac{k}{r}+2\Rightarrow r\geq\frac{k}{n-k-d+2}=\frac{k}{J}.$$

$$r\geq\Big{\lceil}\frac{k}{J}\Big{\rceil}.$$

$$\overline{r}\geq\Big{\lceil}\frac{k}{J}\Big{\rceil}\Big{(}1-\frac{J\Big{\lceil}\frac{k}{J}\Big{\rceil}-k}{n}\Big{)}.$$

$$\sum _{i=1}^{n-\theta }\text{Loc}(y_i)=\sum _{j=1}^J|{\mathcal{Y}}_j|r_j\ge \sum _{j=1}^J|{\mathcal{Y}}_j|(|{\mathcal{Y}}_j|-1),$$

$$n\overline{r}\geq m_{1}\Big{\lfloor}\frac{k}{J}\Big{\rfloor}+{(n-m_{1})}\Big{\lceil}\frac{k}{J}\Big{\rceil}.$$

$$\bar{r}\geq\alpha\Big{\lceil}\frac{k}{J}\Big{\rceil}+(1-\alpha)\Big{\lfloor}\frac{k}{J}\Big{\rfloor},$$

$$\frac{1}{n}\Big{\lceil}\frac{k}{J}\Big{\rceil}\Big{(}J\Big{\lceil}\frac{k}{J}\Big{\rceil}-k\Big{)}=\left\{\begin{array}[]{l l}\frac{1}{n}(J-\beta)\lceil\frac{k}{J}\rceil,&\quad J\nmid k\\ 0,&\quad J\mid k\end{array}\right.$$

$$R=\frac{k}{n}>\Big{(}1-\frac{1}{\sqrt{n}}\Big{)}^{2}.$$

$$\small\overline{r}\geq\frac{\underset{\theta\in[0,d-2]}{\min}\Bigg{\{}(J-a_{\theta})\Big{\lfloor}\frac{n-\theta}{J}\Big{\rfloor}^{2}+a_{\theta}\Big{\lceil}\frac{n-\theta}{J}\Big{\rceil}^{2}+(n-dJ+2J)\theta\Bigg{\}}}{n}-1,$$

$$n\text{leavevmode}\overline{r}=\sum _{i=1}^n\text{Loc}(y_i)=\sum _{j=1}^m|{\mathcal{Y}}_j|r_j,\text{leavevmode}\text{leavevmode}m\in [2,n-k].$$

$$\bigg{\lfloor}\frac{d-2}{\lceil\frac{k}{J}\rceil+1}\bigg{\rfloor}\geq\Big{\lceil}\frac{k}{J}\Big{\rceil}-\theta,$$

$$\theta=(d-2)\mod(\Big{\lceil}\frac{k}{J}\Big{\rceil}+1).$$

$$r_{J+1}=(J-a_{\theta^{*}})\Big{(}\Big{\lfloor}\frac{n-{\theta^{*}}}{J}\Big{\rfloor}-d+2\Big{)}+a_{\theta^{*}}\Big{(}\Big{\lceil}\frac{n-\theta^{*}}{J}\Big{\rceil}-d+2\Big{)}+\theta^{*}-1.$$

$$r_{J+1}=n-J(d-2)-1.$$

$$\displaystyle\begin{split}n\overline{r}_{\mathcal{C}_{3}}&=(J-a_{\theta^{*}})\Big{\lfloor}\frac{n-{\theta^{*}}}{J}\Big{\rfloor}\Big{(}\Big{\lfloor}\frac{n-{\theta^{*}}}{J}\Big{\rfloor}-1\Big{)}\\ &+a_{\theta^{*}}\Big{\lceil}\frac{n-{\theta^{*}}}{J}\Big{\rceil}\Big{(}\Big{\lceil}\frac{n-{\theta^{*}}}{J}\Big{\rceil}-1\Big{)}+\theta^{*}(n-J(d-2)-1)\\ &=(J-a_{\theta}^{*})\Big{\lfloor}\frac{n-\theta^{*}}{J}\Big{\rfloor}^{2}+a_{\theta}^{*}\Big{\lceil}\frac{n-\theta^{*}}{J}\Big{\rceil}^{2}\\ &-\Big{(}a_{\theta^{*}}\Big{\lceil}\frac{n-{\theta^{*}}}{J}\Big{\rceil}+(J-a_{\theta^{*}})\Big{\lfloor}\frac{n-{\theta^{*}}}{J}\Big{\rfloor}\Big{)}\\ &+\theta^{*}(n-J(d-2)-1).\end{split}$$

$$a_{\theta^{*}}\Big{\lceil}\frac{n-{\theta^{*}}}{J}\Big{\rceil}+(J-a_{\theta^{*}})\Big{\lfloor}\frac{n-{\theta^{*}}}{J}\Big{\rfloor}=n-\theta^{*}.$$

$$n\overline{r}_{\mathcal{C}_{3}}=(J-a_{\theta^{*}})\Big{\lfloor}\frac{n-{\theta^{*}}}{J}\Big{\rfloor}^{2}+a_{\theta^{*}}\Big{\lceil}\frac{n-\theta^{*}}{J}\Big{\rceil}^{2}+\theta^{*}(n-J(d-2))-n,$$

$$\begin{split}\mathbf{G}_{0}^{T}=\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ &\\ \left(\begin{array}[]{@{}*{16}{c}@{}}0&0&1&0&0&0&0&0&0&0\\ 0&0&0&1&0&0&0&0&0&0\\ 0&0&0&0&0&0&1&0&0&0\\ 0&0&0&0&0&0&0&0&0&1\\ 35&134&39&29&15&191&187&3&102&38\\ 34&135&39&29&15&191&187&3&102&38\\ 234&137&29&254&245&110&153&9&223&2\\ 243&249&60&11&59&234&48&37&217&104\\ 25&112&32&245&206&132&169&44&6&106\\ 0&0&0&1&1&1&1&0&0&0\\ 1&0&0&0&0&0&0&0&0&0\\ 0&1&0&0&0&0&0&0&0&0\\ 0&0&0&0&1&0&0&0&0&0\\ 0&0&0&0&0&1&0&0&0&0\\ 0&0&0&0&0&0&0&1&0&0\\ 0&0&0&0&0&0&0&0&1&0\\ \end{array}\right),\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \end{split}$$

$$\begin{split}&\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ d-3<\Big{\lceil}\frac{k}{n-k-d+2}\Big{\rceil}\\ &\Leftrightarrow(n-k-d+2)(d-3)<k\\ &\Leftrightarrow(d-2)^{2}-(n-k+1)(d-2)+n>0\\ &\Leftrightarrow(\frac{n-k+1}{2})^{2}-(n-k+1)(\frac{n-k+1}{2})+n>0\\ &\Leftrightarrow 4n-(n-k+1)^{2}>0\\ &\Leftrightarrow\frac{k}{n}>(1-\frac{1}{\sqrt{n}})^{2}\end{split}$$

$$\begin{array}{rl}& \text{leavevmode}\text{leavevmode}\text{leavevmode}\sum _{l=1}^{J-1}|{\mathcal{Y}}_l|+(|{\mathcal{Y}}_J|-{\theta }_i+\theta -d+2-1)+{\theta }_i\\ & =\sum _{l=1}^J|{\mathcal{Y}}_l|+\theta -d+1=(n-\theta )+\theta -d+1\\ & =J+k-1\end{array}$$

$$f_{\min }(m):=f(z_1^{\ast },\cdots ,z_m^{\ast },m)=\underset{z_i}{\min }\text{leavevmode}f(z_1,\cdots ,z_m,m).$$

$$\begin{array}{rl}{f}_{\min }(m)& =\sum _{i=1}^m{w}_i^2=\sum _{i=1}^{m-1}{w}_i^2+(\frac{w_m}{2}+\frac{w_m}{2}{)}^2\\ & \ge \sum _{i=1}^{m-1}{w}_i^2+(\frac{w_m}{2}{)}^2+(\frac{w_m}{2}{)}^2\\ & \ge \sum _{i=1}^{m+1}{v}_i^2=f_{\min }(m+1),\end{array}$$

$$n\text{leavevmode}\overline{r}=\sum _{i=1}^m|{\mathcal{Y}}_i|r_i\ge \sum _{i=1}^m|{\mathcal{Y}}_i|(|{\mathcal{Y}}_i|-1)=\sum _{i=1}^m|{\mathcal{Y}}_i{|}^2-\sum _{i=1}^m|{\mathcal{Y}}_i|$$

Full text

I Introduction

I-A Motivation

Distributed storage systems (DSSs) provide a reliable, flexible, and cost-effective solution for digital data storage, and allow for anytime/anywhere access to one’s data. Reliability and availability are achieved through redundancy. For example, in the commonly used 3-replication method, three replicas of data are stored across distinct data nodes (DNs) [1, 2]. Considering the rapid growth of data as well as costly maintenance of storage components in DSSs, the replication method is becoming unattractive due to its very large storage overhead. Very recently, erasure codes have been used in DSSs to provide redundancy more efficiently [3, 1, 2].

An $(n,k,d)$ linear erasure code maps $k$ information symbols to $n$ encoded symbols $y_{1}$ to $y_{n}$ in a finite field $\mathbb{F}_{q}$, and has minimum distance $d$. The upper bound on $d$ is obtained as $d\leq n-k+1$. Erasure codes that achieve this bound are called maximum distance separable (MDS). A downside of using conventional erasure codes in DSSs is their high-traffic data recovery [2]. For example, in order for a scaler MDS code to recover a missing block, $k$ other encoded symbols have to be downloaded from other DNs. This problem that can be mitigated by using locally repairable codes (LRCs). LRCs are designed such that recovering a lost/erased symbol requires access to a small number of other available symbols [4, 5]. The minimum number of symbols required to construct the $i$-th encoded symbol ($y_{i}$) defines its locality, and is denoted by Loc$(y_{i})$. The code locality ($r$) is defined as the maximum locality of its symbols. Also, the code average locality ($\overline{r}$) is defined as the average of Loc$(y_{i})$, i.e. $\overline{r}=\frac{1}{n}\sum_{i=1}^{n}\text{Loc}(y_{i})$. LRCs have been recently used in Facebook HDFS RAID [2] and Windows Azure Storage [6].

I-B Related Work

The connection between the code locality and other code parameters has been the subject of some recent studies [7, 4]. In [7], it is shown that $d\leq n-k-\lceil\frac{k}{r_{inf}}\rceil+2$, where $r_{inf}$ is the maximum locality of the information symbols. In [4], it is verified that

[TABLE]

where $r$ is the maximum locality of all symbols, i.e. $r=\underset{i\in[1,n]}{\max}\{\text{Loc}(y_{i})\}$. LRCs that achieve the bound in (1) are called optimal.

In [8], optimal LRCs have been proposed over sufficiently large finite fields for the case that $(r+1)|n$. In [9, 10, 11, 12] binary LRCs are proposed. In [13], a bound on $k$ is obtained in terms of $n$, $k$, and $r$ as well as finite field order $q$. In [14], LRCs for small values of $r$ have been proposed some of which reach the bound in [13]. In [15], a tight bound on $d$ is obtained which improves the bound in (1) for some specific values of $n$, $k$, and $r$.

In some other schemes of LRCs, a missing block can be reconstructed by accessing multiple disjoint groups of other blocks, e.g. see [16, 17, 18, 19] and references therein. Other approaches in designing erasure codes for DSSs include reducing the repair bandwidth [20] and disk I/O [21].

I-C Contribution of the Paper

As shown in (5), the bound in (1) is translated to a bound on the maximum locality $r$ in terms of $n$, $k$, and $d$. The aim of this paper, however, is to $i)$ obtain lower bounds on $\overline{r}$ in terms of $n$, $k$, and $d$; and $ii)$ design LRCs that achieve the obtained bounds. In [22], as the first step towards establishing a bound on $\overline{r}$ and motivated by $(n,k,d)=(16,10,5)$ code used in Facebook HDFS-RAID [2], we proved that the average locality $\overline{r}$ of any $(n,k)=(16,10)$ LRC with minimum distance $d=5$ is at least 3.875. We also gave the construction of the LRC that achieves the bound $\overline{r}=3.875$. In this paper, we generalize our work in [22]. More specifically, we establish a lower bound on $\overline{r}$ of arbitrary $(n,k,d)$ LRCs (Theorem 5). Furthermore, we obtain a tight lower bound on $\overline{r}$ for a practical case where $R=\frac{k}{n}>(1-\frac{1}{\sqrt{n}})^{2}$ (Theorem 6). Finally, we design three classes of LRCs that achieve the obtained bounds on $\overline{r}$ (Section V). Note that the improvement achieved by our proposed LRCs comes without sacrificing such crucial coding parameters as rate $(\frac{k}{n})$ and minimum distance $d$. The following simple example shows the effectiveness of our solution.

Example 1

The existing solution: For an $(n,k,r)=(8,4,3)$ LRC, bound in (1) results in $d\leq 8-4-\lceil\frac{4}{3}\rceil+2=4$. In this case, since $(r+1)\mid n$, an optimal LRC with $d=4$ can be constructed whose Tanner graph is depicted in Fig. 1(a). Here, Loc$(y_{i})=r=\overline{r}=3$ for $i\in[1,8]$.

Our proposed solution: Fig. 1(b) shows Tanner graph associated with our proposed LRC in this paper. For this LRC, Loc$(y_{i})=2$ for $i\in[1,6]$ and Loc$(y_{i})=3$ otherwise. Hence, $r=3$ and $\overline{r}=(6\times 2+2\times 6)/8=2.25$. In other words, the average locality is improved by 25% without changing $d$, and the code rate $k/n$.

I-D Importance of The Results

In practice, LRCs are of significant importance as they decrease the costly overhead associated with accessing nodes during a repair process [7]. Also, the average locality of an LRC directly affects the repair bandwidth and disk I/O [2]. To show this, assume that $S_{DN}$ and $S_{Blk}$ represent size of a DN and size of a data block in terms of capacity unit, respectively. Then, each DN can store $\frac{S_{DN}}{S_{Blk}}$ blocks of data. When a DN fails, on average, $\overline{r}\frac{S_{DN}}{S_{Blk}}$ blocks each of size $S_{Blk}$ have to be downloaded from active DNs in order to perform the repair process. In other words, the repair bandwidth is equal to $\overline{r}\frac{S_{DN}}{S_{Blk}}\times S_{Blk}=\overline{r}S_{DN}$. Considering the large capacity of DNs (order of a few TB) as well as restricted repair bandwidth, even a small reduction in $\overline{r}$ can be of great importance [2].

Furthermore, mean-time to data-loss (MTTDL) associated with an LRC is inversely proportional to the average locality of the code111MTTDL is a reliability metric defined as the average time taken such that a stripe of data cannot be recovered due to DN failures. [12]. In other words, for two LRCs with the same values of $(n,k,d)$ and different values of $\overline{r}$, the one with a smaller value of $\overline{r}$ is more reliable in terms of MTTDL.

In this work, by using a novel approach, we design LRCs with improved average locality. In other words, the amount of costly repair bandwidth, disk I/O, number of participating nodes during repair process of a lost/erased block of data, and MTTDL are all improved by using our proposed coding schemes. Considering the recent interest in using efficient erasure codes in real-world data centers (e.g. in Google File System [3], Windows Azure Storage [6], and Facebook HDFS-RAID [2]), our results can be of great importance from a practical point of view.

Notations: We show matrices and vectors by capital boldface letters and boldface letters, respectively. $\mathbb{F}_{q}$ stands for a finite field with cardinality $q$. $\mathbf{I}_{a}$ and $(\cdot)^{T}$ represent an identity matrix of size $a$ and the matrix transpose operation, respectively. For integers $a$ and $b$ with $b\geq a$, $[a,b]=\{a,a+1,\cdots,b\}$. $\overline{\mathcal{A}}$ and $|\mathcal{A}|$ stand for the complement and cardinality of set $\mathcal{A}$, respectively. For two sets $\mathcal{A}$ and $\mathcal{B}$, $\mathcal{A}\backslash\mathcal{B}$ stands for the relative complement of $\mathcal{B}$ in $\mathcal{A}$, i.e., $\mathcal{A}\backslash\mathcal{B}=\{a\in\mathcal{A}|a\notin\mathcal{B}\}$. Finally, $\lfloor\cdot\rfloor$ and $\lceil\cdot\rceil$ represent the floor and ceiling operators, respectively.

II Backgrounds and Definitions

Definition 1

(Systematic Linear Block Codes). The generator matrix of an $(n,k,d)$ systematic linear block code $\mathcal{C}$ can be represented as $\mathbf{G}=[\mathbf{I}_{k},\mathbf{P}]\in{\mathbb{F}_{q}^{k\times n}}$, where $\mathbf{P}\in{\mathbb{F}_{q}^{k\times(n-k)}}$. Hence, the encoded symbols can be obtained by $\mathbf{y}=\mathbf{x}\mathbf{G}$, where $\mathbf{x}=[x_{1},x_{2},...,x_{k}]\in{\mathbb{F}_{q}^{1\times k}}$ and $\mathbf{y}=[y_{1},y_{2},...,y_{n}]\in{\mathbb{F}_{q}^{1\times n}}$ represent the information and encoded vectors, respectively. The parity check matrix of $\mathcal{C}$ is $\mathbf{H}=[-\mathbf{P}^{T},\mathbf{I}_{n-k}]\in{\mathbb{F}_{q}^{(n-k)\times n}}$. For any encoded vector $\mathbf{y}$ of $\mathcal{C}$, we have $\mathbf{y}\mathbf{H}^{T}=\mathbf{0}$. Every $d-1$ columns of the parity check matrix of an $(n,k,d)$ linear block code are independent.

Definition 2

(Minimum distance of code). Assume that $\mathbf{u}$ and $\mathbf{v}$ denote two arbitrary distinct codewords of a code $\mathcal{C}$. Then, the minimum distance of $\mathcal{C}$ can be defined as $d=\min\{d(\mathbf{u},\mathbf{v})\}$, where $d(\mathbf{u},\mathbf{v})$ is the Hamming distance between $\mathbf{u}$ and $\mathbf{v}$. In other words, $d$ is the minimum number of differences between any two codewords of $\mathcal{C}$. For an $(n,k)$ code with the minimum distance of $d$, any $d-1$ symbol erasures can be reconstructed.

Definition 3

(Tanner graph). Tanner graph of an $(n,k)$ linear block code is a bipartite graph with two sets of vertices: a set of $n$ variable nodes and a set of $n-k$ check nodes. The $i$-th variable node for $i\in[1,n]$ is adjacent with the $j$-th check node for $j\in[1,n-k]$ iff $\mathbf{h}_{j}(i)\neq 0$, where $\mathbf{h}_{j}$ represents the $j$-th row of the parity check $\mathbf{H}\in\mathbb{F}_{q}^{(n-k)\times k}$. Therefore, variable nodes linked to a check node are linearly dependent.

In a DSS, linear block codes can be used to encode data as follows. Assume that a stripe of data of size $L_{S}$ symbols in $\mathbb{F}_{q}$ is to be stored in a DSS. The stripe is first broken into $k$ data blocks with size $l_{S}=\frac{L_{S}}{k}$ symbols. We denote by $x_{i,j}$ the $i$-th symbol of the $j$-th block. The coded vector is now obtained by $\mathbf{y}_{i}=[y_{i,1},y_{i,2},...,y_{i,n}]=\mathbf{x}_{i}\mathbf{G}\in\mathbb{F}_{q}^{1\times n}$, where $\mathbf{x}_{i}=[x_{i,1},x_{i,2},...,x_{i,k}]\in\mathbb{F}_{q}^{1\times k}$ with $i\in[1,l_{S}]$. The $l_{S}$ encoded vectors $\mathbf{y}_{i}$’s are then stacked to constitute $\mathbf{Y}$ whose columns are $n$ encoded blocks stored in $n$ distinct DNs. For the sake of simplicity, we assume $l_{S}=1$ from now on.

Definition 4

(Locality). We denote the locality of an encoded symbol $y_{i}$ by Loc$(y_{i})$ and define it as the minimum number of elements of $\mathcal{Y}\setminus\{y_{i}\}$ whose linear combination form $y_{i}$, where $\mathcal{Y}=\{y_{1},...,y_{n}\}$. Therefore, there exists a subset of $\mathcal{Y}\setminus\{y_{i}\}$ indexed by $\mathcal{I}\subseteq[1,n]$ such that $|\mathcal{I}|=Loc(y_{i})$ and

[TABLE]

where Loc$(y_{i})$ is the smallest possible integer and $\Theta=\{i\}\cup\mathcal{I}$. The average and maximum locality of the code are defined as $\overline{r}=\frac{\sum_{i=1}^{n}\text{Loc}(y_{i})}{n}$ and $r=\underset{i\in[1,n]}{\max}\{\text{Loc}(y_{i})\}$, respectively.

III Preliminary Results

In this section, we present definitions and lemmas required in discussing and proving our main results. First, we show that there exists a Tanner graph, called locality Tanner graph, with at most $n-k$ CNs which determines the locality of all the encoded symbols. Then, after defining and constructing local groups, we establish lower bounds on the average locality $\overline{r}$ in Section IV.

A code does not have a unique Tanner graph representation. In the following lemma, we define a representation of a Tanner graph, called locality Tanner graph, which reflects the locality of all the encoded symbols.

Lemma 1

*A Tanner graph is called a locality Tanner graph, if there is a subset of CNs $\mathcal{P}_{L}\subseteq\mathcal{P}$, such that

(i) any $y_{i}\in\mathcal{Y}$ is connected to at least one CN in $\mathcal{P}_{L}$ with degree $\text{Loc}(y_{i})+1$;

(ii) check nodes in $\mathcal{P}_{L}$ are linearly independent (consequently, $|\mathcal{P}_{L}|\leq n-k$).

CNs in $\mathcal{P}_{L}$ are called local CNs; others are called global CNs. The set of VNs adjacent to a local CN is called a local group.*

Proof:

Our proof is by construction. First, a local group $\Psi_{1}$ with the minimum locality is selected from $\mathcal{Y}=\overline{\mathcal{U}_{1}}$, where $\mathcal{U}_{1}:=\{\}$. Then, a local group $\Psi_{2}$ with the minimum locality is selected from $\overline{\mathcal{U}_{2}}$, where $\mathcal{U}_{2}=\mathcal{U}_{1}\cup\Psi_{1}$. Similarly, a local group $\Psi_{i}$ with the minimum locality is selected from $\overline{\mathcal{U}_{i+1}}$, where $\mathcal{U}_{i+1}=\mathcal{U}_{i}\cup\Psi_{i}$. Note that each local group represents a CN. Since there are at most $n-k$ linearly independent equations of form (2), the algorithm terminates at most after $n-k$ local groups are selected, i.e. $|\mathcal{P}_{L}|\leq n-k$. ∎

Local group construction: By using a greedy algorithm, we partition the set of the encoded symbols $\mathcal{Y}=\{y_{1},...,y_{n}\}$ into local groups $\mathcal{Y}_{j}$’s for $j\in[1,m]$ such that all elements of $\mathcal{Y}_{j}$ have the same locality $r_{j}$, where $r_{1}\leq r_{2}\leq\cdots\leq r_{m}$. The detailed procedure is described in the following algorithm.

Definition 5

(Non-overlapping local groups). We say that local groups one to $a$ are non-overlapping if intersection between any two of $\mathcal{Y}_{1}$ to $\mathcal{Y}_{a}$ is empty, i.e. $\mathcal{Y}_{s}\bigcap\mathcal{Y}_{w}=\emptyset$ for any distinct $s$ and $w$ in $[1,a]$.

Example 2

Fig. 2 shows a locality Tanner graph with $n=16$ VNs and $m=4$ local CNs. In this figure, global CNs are not shown. This locality Tanner graph has four local groups $\mathcal{Y}_{1}$ to $\mathcal{Y}_{4}$.

The following Lemma shows how Tanner graph of an $(n,k)$ linear block code is related to the minimum distance of the code ($d$).

Lemma 2

Over a sufficiently large finite field, a necessary and sufficient condition for an $(n,k)$ linear block code $\mathcal{C}$ with Tanner graph $\mathcal{T}$ to have minimum distance $d$ is that every $\gamma$ CNs of $\mathcal{T}$ cover at least $\gamma+k$ VNs for every $\gamma\in[J,\leavevmode\nobreak\ n-k]$, where $J:=n-k-d+2$.

Proof:

Necessary condition (assumption: the minimum distance is $d$): By contradiction, assume that there is a subset of CNs of $\mathcal{T}$ with cardinality $\gamma$ covering at most $\gamma+k-1$ VNs. Also, assume that the $n-(\gamma+k-1)\in[1,d-1]$ VNs not covered by the mentioned $\gamma$ CNs are erased. Observe that among all $\gamma+k-1$ VNs covered by the mentioned $\gamma$ CNs, there are at most $(\gamma+k-1)-\gamma=k-1$ independent VNs. This implies that $\mathcal{C}$ cannot recover $k$ information symbols for the assumed $n-(\gamma+k-1)\in[1,d-1]$ erasures. This contradicts the fact that an erasure code with minimum distance $d$ can recover $k$ information symbols for any up to $d-1$ symbol erasures.

Sufficient condition (assumption: every $\gamma$ CNs of $\mathcal{T}$ cover at least $(\gamma+k)$ VNs for every $\gamma\in[J,\leavevmode\nobreak\ n-k]$): An erasure code with minimum distance $d$ can recover any up to $d-1$ erasures. Let us define $\beta$ as $\beta:=(n-k)-\gamma+1$. Since $\gamma\in[n-k-d+2,\leavevmode\nobreak\ n-k]$, we have $\beta\in[1,d-1]$. Now, we show that any $\beta$ VNs are covered by at least $\beta$ CNs. Over a sufficiently large finite field $\mathbb{F}_{q}$, this implies that any $\beta\in[1,d-1]$ erasures can be recovered using $\beta$ independent equations associated with their corresponding CNs. By contradiction, assume that there is a subset of VNs with cardinality $\beta$ covered by at most $\beta-1$ VNs. Then, the remaining $(n-k)-(\beta-1)=\gamma$ CNs cover at most $n-\beta=k+\gamma-1$ VNs. This contradicts the assumption that any $\gamma$ CNs cover at least $k+\gamma$ VNs. ∎

Remark 1

As we will see, all LRCs discussed in this paper have the following two properties: $i)$ each local CN is connected to at least one VN which is not connected to other local CNs, and $ii)$ each global CN is linked to all $n$ VNs. When these conditions are satisfied, Lemma 2 results in the following corollary which allows us to verify that the minimum distance of a code is $d$ by showing that every $J$ local CNs cover at least $J+k$ VNs.

Corollary 3

For an LRC with Tanner graph $\mathcal{T}$, assume that i) each local CN is connected to at least one VN which is not connected to other local CNs, and ii) each global CN is linked to all $n$ VNs. Then, the sufficient condition of Lemma 2 is equivalent to the following one. If every $J$ local CNs cover $J+k$ VNs, then the minimum distance of the code is $d$.

Proof:

Without loss of generality, consider the first $J$ local CNs that cover at least $J+k$ VNs. By adding $\gamma-J$ arbitrary local CNs to these $J$ local CNs, at least $\gamma-J$ more VNs are covered, where $\gamma\in[J,n-k]$. This is true because each local CN is connected to at least one VN which is not connected to other local CNs. Also, each global CN is connected to all VNs. Hence, every $\gamma$ CNs cover at least $\gamma+J$ VNs. ∎

As we will see later, Lemma 2 is a strong tool which will help us obtain a bound on $\overline{r}$. In fact, the famous bound of (1) on the maximum locality also can easily be obtained by Lemma 2.

Remark 2

*For an $(n,k,d,r)$ LRC, assume that the number of local CNs is $m$, where $m\in\{2,\cdots,J,J+1,\cdots,n-k\}$. Now, we consider two cases as follows.

Case (i) $m\in[J,n]$: In this case, by Lemma 2, every $\gamma$ local CNs cover at least $\gamma+k$ VNs. We have*

[TABLE]

Case (ii) $m\in[2,J-1]$: In this case, all the $m$ local CNs cover all the $n$ VNs. We have

[TABLE]

where the last equality holds because $n\geq J+k=n-(d-2)$. From (3), we have

[TABLE]

Since $d$ is an integer, d\leq n-k-\Big{\lceil}\frac{k}{r}\Big{\rceil}+2.

The following Lemma is used to partition a subset of the encoded symbols, say $\mathcal{A}$, with cardinality $A$ into $\zeta$ non-overlapping local groups such that the average locality of the encoded symbols in $\mathcal{A}$ is minimized.

Lemma 4

Let $z_{j}$, $j\in[1,\zeta]$ be integers, and $\sum_{j=1}^{\zeta}z_{j}=A$. Then,

[TABLE]

where $a=A+\zeta-\zeta\lceil\frac{A}{\zeta}\rceil$.

Proof:

Subject to $\sum_{j=1}^{\zeta}z_{j}=A$ and integer $z_{j}$, the sum $\sum_{j=1}^{\zeta}z_{j}^{2}$ is minimized if $|z_{l}-z_{m}|\leq 1$ for every pair $z_{l}$ and $z_{m}$. Because otherwise if $z_{l}>z_{m}+1$, then $\sum_{j=1}^{\zeta}z_{j}^{2}$ can be reduced by setting $z_{l}$ to $z_{l}-1$ and $z_{m}$ to $z_{m}+1$. This is true since

[TABLE]

Consequently, $\sum_{j=1}^{\zeta}z_{j}^{2}$ is minimized if for every $j$, $z_{j}=z$ or $z_{j}=z-1$ for some integer $z$. The number $z$ is unique and it is a function of $A$ and $\zeta$. Now, assume that among $\zeta$ integers $z_{j}$, $a$ integers are $z$ and the rest $\zeta-a$ integers are $z-1$. Therefore, $az+(\zeta-a)(z-1)=A$; equivalently, $a=A+\zeta(1-z)$. Hence, $\lceil\frac{a}{\zeta}\rceil=\lceil\frac{A}{\zeta}\rceil+1-z$. Therefore, $z=\lceil\frac{A}{\zeta}\rceil$ because $a\leq\zeta$. ∎

IV Lower Bounds on $\overline{r}$

First, in Section IV-A, we derive a lower bound on $\overline{r}$ that holds for any $(n,k,d)$ LRCs. We compare this bound to the one on maximum locality $r$. In Section IV-B, we obtain a tight lower bound on $\overline{r}$ for $(n,k,d)$ LRCs which satisfy the following constraint on the code rate R>\Big{(}1-\frac{1}{\sqrt{n}}\Big{)}^{2}. In section V, we present three classes of LRCs that achieve the obtained bounds presented in this section.

IV-A A Lower Bound on $\overline{r}$ for Arbitrary $(n,k,d)$ LRC

First, let us reform the bound in (1) to obtain a lower bound on $r$. From (1), we have

[TABLE]

Since $r$ is an integer, the lower bound of the maximum locality $(r)$ can be presented as

[TABLE]

Theorem 5

For any $(n,k,d)$ LRC with $J=n-k-d+2$, the average locality of all symbols ($\overline{r}$) is bounded as follows

[TABLE]

Proof:

First, we construct local groups using Algorithm 1. For the code to have the minimum distance of $d$, by Lemma 2, the first $J$ local groups must cover $n-\theta$ VNs for some integer $\theta\in[0,d-2]$.

[TABLE]

where $\sum_{j=1}^{J}|\mathcal{Y}_{j}|=n-\theta$. Observe that the minimum of $\sum_{j=1}^{J}|\mathcal{Y}_{j}|(|\mathcal{Y}_{j}|-1)$ is obtained when $\sum_{j=1}^{J}|\mathcal{Y}_{j}|$ is minimized, i.e. when $\sum_{j=1}^{J}|\mathcal{Y}_{j}|=n-(d-2)=J+k.$ In this case, by lemma 4, there are $(J-a_{d-2})$ local groups with cardinality $(\lfloor\frac{k}{J}\rfloor+1)$ and locality $\lfloor\frac{k}{J}\rfloor$; and $a_{d-2}$ local groups with cardinality $(\lceil\frac{k}{J}\rceil+1)$ and locality $\lceil\frac{k}{J}\rceil$, where $a_{d-2}=k+J-J\lceil\frac{k}{J}\rceil$.

Note that according to Algorithm 1, all VNs not in the first $J$ local groups have locality greater than or equal to the maximum locality of the first $J$ local groups. In order to obtain a lower bound on $\overline{r}$, we assume that all the remaining $d-2$ VNs not in the first $J$ local groups have locality equal to $\lceil\frac{k}{J}\rceil$, because for any $\theta$, we have$\max\limits_{j\in[1,J]}r_{j}\geq\lceil\frac{k}{J}\rceil$. Hence, the minimum of $\overline{r}$ is achieved if there are $m_{1}:=(J-a)(\lfloor\frac{k}{J}\rfloor+1)$ VNs with locality $\lfloor\frac{k}{J}\rfloor$ and $n-m_{1}$ VNs with locality $\lceil\frac{k}{J}\rceil$. In other words,

[TABLE]

If $J\mid k$, then by replacing $a=J$ and $m_{1}=0$ in (7), we have $\overline{r}\geq\lceil\frac{k}{J}\rceil$. If $J\nmid k$, then by replacing $m_{1}=(J\lceil\frac{k}{J}\rceil-k)\lceil\frac{k}{J}\rceil$ in (7), we have $\overline{r}\geq\lceil\frac{k}{J}\rceil(1-\frac{m_{1}}{n})$. ∎

Remark 3

Considering (7), the bound on $\overline{r}$ can be represented as follows

[TABLE]

*where $\alpha=\frac{1}{n}(J\lceil\frac{k}{J}\rceil-k)(\lfloor\frac{k}{J}\rfloor+1)$. *

Remark 4

The gap between the bounds on $r$ and $\overline{r}$, represented in (5) and (6) respectively, is maximized when $J\mid(k-1)$. To see this, observe that by subtracting the right hand side of (6) from that of (5), we have

[TABLE]

where $\beta=k\mod J$, and $\beta\neq 0$. Hence, (9) achieves its maximum value when $\beta=1$, equivalently, when $J\mid(k-1)$. In Section V-B, we show that for sufficiently large values of $d$, the bound in Theorem 5 is achievable when $J\mid(k-1)$.

IV-B A Tight Lower Bound on $\overline{r}$ of LRCs with R>\Big{(}1-\frac{1}{\sqrt{n}}\Big{)}^{2}

In the previous section, in order to obtain a lower bound on $\overline{r}$, we assumed that the first $J$ local groups of $(n,k,d)$ LRCs cover $J+k$ VNs. Also, we assumed that locality of the remaining $n-(J+k)=d-2$ is equal to maximum locality of the first $J+k$ VNs. The obtained bound in Theorem 5 is not always tight. In this section, we obtain a tight lower bound on $\overline{r}$ assuming that

[TABLE]

It is notable that linear block codes currently in use—e.g. the $(16,10,5)$ code used in Facebook HDFS-RAID, the $(16,12,4)$ code used in Windows Azure Storage, and the $(9,6,4)$ code in Google File System—satisfy the condition presented in (10).

Theorem 6

For any $(n,k,d)$ LRC with R=\frac{k}{n}>\big{(}1-\frac{1}{\sqrt{n}}\big{)}^{2}, the average locality of all symbols ($\overline{r}$) is bounded as follows

[TABLE]

where $J=n-k-d+2$ and a_{\theta}=n-\theta+J-J\Big{\lceil}\frac{n-\theta}{J}\Big{\rceil}.

Proof:

Here, we present the sketch of the proof. Please find the detailed proof in Appendix VIII.

To begin with, we construct $m$ local groups using Algorithm 1. Note that the maximum locality of an $(n,k)$ linear block code is $k$, hence, $m>1$. Furthermore, by Lemma 1, $m\leq n-k$. Therefore,

[TABLE]

Now, we consider two cases. As the first case, we assume that the total number of local groups is less than or equal to $J$, i.e. $m\in[2,J]$. In this case, we verify that the minimum average locality is achieved when $m=J$.

As the second case, we assume that the total number of local groups is greater than $J$, thus $m\in[J+1,n-k]$. In this case, we verify that the minimum average locality is obtained when there are $m=J+1$ local groups. By Lemma 2, in order to achieve minimum distance $d$, the first $J$ local groups must cover at least $J+k=n-(d-2)$ VNs and at most $n-1$ VNs 222Note that the case that the first $J$ local groups cover $n$ VNs is equivalent to the first case with $m=J$. This case is considered in Theorem 6 by setting $\theta=0$.. Among all possible $J+1$ local groups, we assume that the first $J$ local groups and the last local group cover $n-\theta$ and $\theta$ VNs, respectively, where $\theta\in[1,d-2]$. Then, we obtain a lower bound on $\overline{r}$ according to this assumption. Finally, the lower bound on $\overline{r}$ is obtained by taking the minimum of two bounds associated with the considered cases. ∎

In Section V-C, we design a class of LRCs that achieve the bound on $\overline{r}$ obtained in Theorem 6.

V Achieving the Bounds: $\overline{r}$-Optimal LRCs

In this section, we design LRCs that achieve the lower bounds on $\overline{r}$ obtained in Theorems 5 and 6.We call such codes, $\overline{r}$-optimal LRCs. In Sections V-A and V-B, we construct two classes of $\overline{r}$-optimal LRCs that achieve the general bound in Theorem 5. Furthermore, in Section V-C, we construct a class of LRCs that achieve the bound in Theorem 6.

V-A The First Class of $\overline{r}$-Optimal LRCs

Suppose the parameters $n$, $k$ and $d$ are such that $(\lceil\frac{k}{J}\rceil+1)\mid(d-2)$. Then, the first class of our proposed $\overline{r}$-optimal LRCs ($\mathcal{C}_{\overline{r}LRC_{1}}$) achieving the bound in Theorem 5 can be constructed as follows. The first $k+J$ VNs are partitioned into $J\lceil\frac{k}{J}\rceil-k$ and $J+k-J\lceil\frac{k}{J}\rceil$ local groups with cardinality $\lfloor\frac{k}{J}\rfloor+1$ and $\lceil\frac{k}{J}\rceil+1$, respectively. Then, $n-(J+k)=d-2$ VNs not covered by the first $J$ local CNs are partitioned into $\frac{d-2}{\lceil\frac{k}{J}\rceil+1}$ local groups with cardinality $\lceil\frac{k}{J}\rceil+1$. Hence, there are exactly $\frac{m_{1}}{\lfloor\frac{k}{J}\rfloor+1}=J\lceil\frac{k}{J}\rceil-k$ and $\frac{n-m_{1}}{\lceil\frac{k}{J}\rceil+1}$ local groups with cardinality $\lfloor\frac{k}{J}\rfloor+1$ and $\lceil\frac{k}{J}\rceil+1$, respectively. Observe that for this class of $\overline{r}$-optimal LRCs, the constraint on $d$ expressed in Corollary 3 is satisfied and the bound in Theorem 5 is achieved.

V-B The Second Class of $\overline{r}$-Optimal LRCs

Suppose the parameters $n$, $k$ and $d$ are such that $J\mid(k-1)$ and

[TABLE]

where

[TABLE]

Fig. 3 shows the Tanner graph corresponding to our second class of $\overline{r}$-optimal LRCs ($\mathcal{C}_{\overline{r}LRC_{2}}$). In this graph, all the local groups, except $\lceil\frac{k}{J}\rceil+2-\theta$, are non-overlapping. Among the non-overlapping local groups, $J-1$ groups have cardinality $\lfloor\frac{k}{J}\rfloor+1$, while the remaining have cardinality $\lceil\frac{k}{J}\rceil+1$. Among the overlapping local groups, \Big{\lceil}\frac{k}{J}\Big{\rceil}+1-\theta groups have cardinality $\lceil\frac{k}{J}\rceil+1$, while the remaining one has cardinality $\theta$. This last overlapping local group shares exactly one VN with (\Big{\lceil}\frac{k}{J}\Big{\rceil}+1-\theta) overlapping local groups, which is possible by (13).

Note that each VN has locality of $\lfloor\frac{k}{J}\rfloor$ or $\lceil\frac{k}{J}\rceil$, and the average locality is optimum by Theorem 5. Also, it can be verified that every $J$ local CNs cover at least $J+k$ VNs, thus, by Corollary 3, the minimum distance of $\mathcal{C}_{\overline{r}LRC_{2}}$ is at least $d$.

V-C The Third Class of $\overline{r}$-Optimal LRCs

Suppose the parameters $n$ and $k$ are such that $R>(1-\frac{1}{\sqrt{n}})^{2}$. Then, the third class of our proposed $\overline{r}$-optimal LRCs ($\mathcal{C}_{\overline{r}LRC_{3}}$) achieving the bound in Theorem 6 is constructed through the following steps (Fig. 4).

Step 1) Obtain $\theta\in[0,d-2]$, denoted $\theta^{*}$, that minimizes (11) in Theorem 6.

Step 2) Partition the set of $n$ VNs $\mathcal{Y}=\{y_{1},\cdots,y_{n}\}$ into two subsets $\mathcal{Y}_{A}$ and $\mathcal{Y}_{B}$ with cardinality $n-{\theta^{*}}$ and $\theta^{*}$, respectively.

Step 3) Partition $n-\theta^{*}$ VNs associated with $\mathcal{Y}_{A}$ into $J=n-k-d+2$ local groups as follows: $J-a_{\theta^{*}}$ local groups with cardinality $\lfloor\frac{n-\theta^{*}}{J}\rfloor$; and $a_{\theta^{*}}$ local groups with cardinality $\lceil\frac{n-\theta^{*}}{J}\rceil$, where a_{\theta^{*}}=n-\theta^{*}+J-J\Big{\lceil}\frac{n-\theta^{*}}{J}\Big{\rceil}. Therefore, there are $(J-a_{\theta^{*}})\lfloor\frac{n-\theta^{*}}{J}\rfloor$ VNs with locality \Big{\lfloor}\frac{n-\theta^{*}}{J}\Big{\rfloor}-1 and $a_{\theta^{*}}\lceil\frac{n-\theta^{*}}{J}\rceil$ VNs with locality \Big{\lceil}\frac{n-\theta^{*}}{J}\Big{\rceil}-1.

Step 4) If $\theta^{*}\neq 0$, construct $(J+1)$-th local group with the following VNs: $i)$ $\lfloor\frac{n-{\theta^{*}}}{J}\rfloor-d+2$ VNs from each of local groups $1$ to $J-a_{\theta^{*}}$; $ii)$ $\lceil\frac{n-\theta^{*}}{J}\rceil-d+2$ VNs from each of local groups $J-a_{\theta^{*}}+1$ to $J$; and $iii)$ all the $\theta^{*}$ VNs associated with $\mathcal{Y}_{B}$. Hence, locality of the last local group is

[TABLE]

By manipulation, we have

[TABLE]

Step 5) Observe that there are $J+b_{\theta^{*}}$ local CNs and $n-k-(J+b_{\theta^{*}})$ global CNs, where $b_{\theta^{*}}=0$ if $\theta^{*}=0$ and $b_{\theta^{*}}=1$ otherwise. Connect each of the $n-k-J-b_{\theta^{*}}=d-2-b_{\theta^{*}}$ to all $n$ VNs.

Remark 5

It can be verified that every $J$ local CNs cover at least $J+k$ VNs, thus, by Corollary 3, the minimum distance of $\mathcal{C}_{\overline{r}LRC_{3}}$ is at least $d$.

Example 3

In this example, we construct an $(n,k,d)=(8,4,4)$ LRC by following the aforementioned steps. Here, $J=n-k-d+2=2$ and $\theta$ minimizing (11) is $\theta^{*}=2$. All the eight VNs are partitioned into two sets $\mathcal{Y}_{A}=\{y_{1},\cdots,y_{6}\}$ and $\mathcal{Y}_{B}=\{y_{7},y_{8}\}$. Since $J\mid n-\theta^{*}$, there are $J=2$ local groups with cardinality $\frac{n-\theta^{*}}{J}=3$. Since $\theta^{*}\neq 0$, the third local group has to be constructed by $\lceil\frac{n-\theta^{*}}{J}\rceil-d+2=\lceil\frac{8-2}{2}\rceil-4+2=1$ VN from each of the first two local groups and $\theta^{*}=2$ VNs associated with $\mathcal{Y}_{B}$. Let us form the third local group by VNs $y_{3}$, $y_{6}$, $y_{7}$, and $y_{8}$. There is $n-k-J-b_{\theta^{*}}=8-4-2-1=1$ global check node which is connected to all $8$ VNs. Fig. 1(b) depicts Tanner graph associated with LRC of this example.

Proposition 7

$\mathcal{C}_{\overline{r}LRC_{3}}$* satisfies the bound on $\overline{r}$ in Theorem 6 with equality.*

Proof:

The first $n-\theta^{*}$ VNs of the code are partitioned into $(J-a_{\theta^{*}})\lfloor\frac{n-{\theta^{*}}}{J}\rfloor$ and $a_{\theta^{*}}\lceil\frac{n-{\theta^{*}}}{J}\rceil$ VNs with locality \Big{\lfloor}\frac{n-{\theta^{*}}}{J}\Big{\rfloor}-1 and \Big{\lceil}\frac{n-{\theta^{*}}}{J}\Big{\rceil}-1 , respectively. Also, locality of the last $\theta^{*}$ VNs is $n-J(d-2)-1$. Hence, for the average locality of $\mathcal{C}_{\overline{r}LRC_{3}}$, denoted $\overline{r}_{\mathcal{C}_{3}}$, we have

[TABLE]

By manipulation, we have

[TABLE]

Thus,

[TABLE]

which is equivalent to the minimum of $\overline{r}$ in Theorem 6. ∎

VI Improvement of the LRC Used in Facebook HDFS-RAID

In this section, we show how our proposed approach can improve locality of the $(n,k,d,\overline{r})=(16,10,5,5)$ LRC used in Facebook HDFS-RAID [2], denoted $\mathcal{C}_{F}$, without sacrificing its crucial parameters, namely the code rate $(R)$ and the code minimum distance $(d)$. We show that by using our proposed LRC, the average locality of $\mathcal{C}_{F}$ is improved 22.5%.

For $\mathcal{C}_{F}$, we have $R=0.625>(1-\frac{1}{\sqrt{n}})^{2}=0.56$. Hence, the code construction described in Section V-C can be used. For an $(n,k,d)=(16,10,5)$ LRC, it is verified by Theorem 6 that $\overline{r}\geq 3.875$, which is obtained for $\theta=d-2=3$.

The Tanner graph of our proposed LRC, denoted $\mathcal{C}_{0}$, is presented in Fig 5. By using the Tanner graph, the parity check matrix of $\mathcal{C}_{0}$, denoted $\mathbf{H}_{0}$, is constructed. The non-zero elements of $\mathbf{H}_{0}\in\mathbb{F}_{2^{8}}^{6\times 16}$ are numerically selected from the finite field $\mathbb{F}_{2^{8}}$ formed by the primitive polynomial $x^{8}+x^{4}+x^{3}+x^{2}+1$ such that every $d-1=4$ columns of $\mathbf{H}_{0}$ are linearly independent. Then, by multiplying the inverse of a full-rank $6\times 6$ sub-matrix of $\mathbf{H}_{0}$ from the left by $\mathbf{H}_{0}$, the systematic form of $\mathbf{H}_{0}$ is obtained. By using the obtained systematic form, the systematic form of the generator matrix of our proposed $(n,k,d,\overline{r})=(16,10,5,3.875)$ LRC, denoted $\mathbf{G}_{0}\in\mathbb{F}_{2^{8}}^{10\times 16}$, can be explicitly represented as follows.

[TABLE]

where each element of $\mathbf{G}_{0}$, denoted $g_{i,j}$ for $i\in[1,16]$ and $j\in[1,10]$, is the decimal representation of a byte of the form $[a_{7},\cdots,a_{0}]\in\mathbb{F}_{2}^{1\times 8}$, i.e. $g_{i,j}=\sum_{l=0}^{7}a_{l}2^{l}$ where $a_{l}\in\mathbb{F}_{2}$, $l\in[0,7]$. For example, $35\equiv[0,0,1,0,0,0,1,1]$.

VII Conclusion and Future Work

The average locality of locally repairable codes (LRCs) is directly translated to the costly repair bandwidth of distributed storage systems. While the existing literature mainly focuses on the maximum locality of erasure codes used for distributed storage systems, the average locality $\overline{r}$ is also an important factor which directly affects costly repair bandwidth and disk I/O. In this paper, we used a novel approach to find a general lower bound on $\overline{r}$ of erasure codes with arbitrary parameters. We also proposed LRCs that achieve the obtaining bounds, offering improvement on $\overline{r}$ over the existing locally reparable codes.

Generalizing our proposed lower bound on $\overline{r}$ for vector codes and erasure codes with multiple groups of locality as well as constructing explicit codes with small $\overline{r}$ can be of interest.

VIII Proof of Theorem 6

[Proof of Theorem 6]

Lemma 8

For an $(n,k,d)$ code with $R>(1-\frac{1}{\sqrt{n}})^{2}$, we have d-3<\Big{\lceil}\frac{k}{n-k-d+2}\Big{\rceil}=\Big{\lceil}\frac{k}{J}\Big{\rceil}.

Proof:

We have

[TABLE]

where the forth inequality is true because the second degree equation $x^{2}-(n-k+1)x+n$ takes its minimum at $x=\frac{n-k+1}{2}$. ∎

Lemma 9

*For a linear block code with minimum distance $d$ and its corresponding locality Tanner graph $\mathcal{T}$, assume that

1. total number of local CNs is greater than $J$, i.e. $m>J$;

2. the first $J$ local CNs of $\mathcal{T}$ cover $(n-\theta)$ VNs, where $\theta\in[1,d-2]$; and

3. among all $\theta$ VNs not covered by the first $J$ local CNs, only $\theta_{i}$ VNs are linked to $i$-th local CN, where $\theta_{i}\in[1,\theta]$ and $i\in[J+1,m]$ (Fig. 6). Then, each local group $\mathcal{Y}_{l}$ with cardinality more than $\theta_{i}$ has at least $(|\mathcal{Y}_{l}|-\theta_{i}+\theta-d+2)$ VNs connected to local CN associated with local group $\mathcal{Y}_{i}$, where $l\in[1,J]$ and $i\in[J+1,m]$.*

Proof:

Without loss of generality, assume that local group $\mathcal{Y}_{J}$ with $|\mathcal{Y}_{J}|>\theta_{i}$ has less than $(|\mathcal{Y}_{J}|-\theta_{i}+\theta-d+2)$ VNs in common with local group $\mathcal{Y}_{J+1}$. This implies that $J$ local CNs indexed by $\{1,2,\cdots,J-1,J+1\}$ cover at most

[TABLE]

VNs which contradicts Lemma 2. ∎

Lemma 10

Consider function $f(z_{1},\cdots,z_{m},m)=\sum_{i=1}^{m}{z_{i}}^{2}$, where $z_{i}$’s are positive non-zero integers for $i\in[1,m]$ which satisfy $\sum_{i=1}^{m}{z_{i}}=A$, and $A$ is a constant integer. Assume that minimum of $f(z_{1},\cdots,z_{m},m)$ is attained when $z_{i}=z_{i}^{*}\leavevmode\nobreak\ \forall i$, i.e.

[TABLE]

Then, $f_{\min}(m)$ is a decreasing function.

Proof:

We need to show that $f_{\min}(m)\geq f_{\min}(m+1)$. Assume that $f_{\min}(m)=f(w_{1},\cdots,w_{m},m)$ and $f_{\min}(m+1)=f(v_{1},\cdots,v_{m+1},m+1)$ with $\sum_{i=1}^{m}w_{i}=\sum_{i=1}^{m+1}v_{i}=A$. We have,

[TABLE]

where the first inequality is by $(a+b)^{2}\geq a^{2}+b^{2}$. ∎

Proof of Theorem 6: By using Algorithm 1, the set of $n$ VNs $\{y_{1},\cdots,y_{n}\}$ is partitioned into $m$ local groups $\mathcal{Y}_{1}$ to $\mathcal{Y}_{m}$. Then, we calculate the average locality in terms of code’s parameters for different values of $m\in[2,n-k]$. More specifically, in order to evaluate $\overline{r}$, we assume that $m\in[2,J]$ and $m\in[J+1,n-k]$ for the first case and the second case, respectively.

case (i) $m\in[2,J]$: In this case,

[TABLE]

Observe that $\sum_{i=1}^{m}|\mathcal{Y}_{i}|=n,\leavevmode\nobreak\ \forall m\in[2,J]$ because all the $m$ local groups have to cover all $n$ VNs. By Lemma 10, $\sum_{i=1}^{m}|\mathcal{Y}_{i}|^{2}$ is a decreasing function. Hence, its minimum is attained when $m=J$. Therefore, considering (17) and Lemma 4, we have

[TABLE]

where a_{0}=n+J-J\big{\lceil}\frac{n}{J}\big{\rceil}. Observe that replacing $\theta$ with zero represents (18) in Theorem 6. Now, we consider the second case where not all $n$ VNs are covered by the first $J$ CNs.

case (ii) $m\in[J+1,n-k]$: Considering Algorithm 1, in this case, $\mathcal{U}_{J+1}=\mathcal{Y}_{1}\bigcup\cdots\bigcup\mathcal{Y}_{J}$. By Lemma 2, every $J$ CNs must cover at least $J+k$ VNs. Hence, the first $J$ local CNs cover at least $J+k$ VNs, i.e. $|\mathcal{U}_{J+1}|=\sum_{i=1}^{J}|\mathcal{Y}_{i}|\in[J+k,n]$. Observe that $\sum_{i=1}^{J}|\mathcal{Y}_{i}|=n$ is equivalent to the first case with $m=J$. In the following, we consider the remaining cases where $J$ local CNs cover $\sum_{i=1}^{J}|\mathcal{Y}_{i}|\in[J+k,n-1]$ VNs. Equivalently, the case where $\theta:=|\overline{\mathcal{U}}_{J+1}|=n-|\mathcal{U}_{J+1}|\in[1,d-2]$ VNs are not covered by the first $J$ local CNs.

Observe that total number of local CNs is $m\in[J+1,n-k]$ among which the first $J$ local CNs determine locality of the first $n-\theta\in[n-(d-2),n-1]$ VNs and the remaining $(m-J)\in[1,d-2]$ local CNs determine locality of $\theta\in[1,d-2]$ VNs not covered by the first $J$ local CNs (Fig. 6). We have

[TABLE]

where $\sum_{i=1}^{J}{|\mathcal{Y}_{i}|}=n-\theta$. Note that in (19), the last inequality holds because locality of $\theta$ VNs not covered by the first $J$ local CNs is determined by the last $m-J$ local CNs.

Now, we obtain $\min\limits_{i\in[J+1,m]}r_{i}$. Assume that among all $\theta$ VNs not covered by local CNs 1 to $J$, only $\theta_{i}$ VNs are linked to $i$-th local CN, where $\theta_{i}\in[1,\theta]$ and $i\in[J+1,m]$. Observe that in order to satisfy Lemma 2, $i$-th CN must have some VNs, denoted $l_{i}$, in common with local groups $1$ to $J$ (Fig.  6). Hence, locality associated with $i$-th local CNs for $i\in[J+1,m]$ is obtained as follows

[TABLE]

Assume that among local CNs 1 to $J$, $\delta_{i}$ local CNs have cardinality more than $\theta_{i}$ and the rest $(J-\delta_{i})$ local CNs have cardinality less than or equal to $\theta_{i}$, with $\delta_{i}\in[1,J-1]$. In other words,

[TABLE]

By Lemma 9, $l$-th local CN with $|\mathcal{Y}_{l}|>\theta_{i}$ must have at least $(|\mathcal{Y}_{l}|-\theta_{i}+\theta-d+2)$ VNs in common with $i$-th local CN, where $l\in[J+1-\delta_{i},J]$ and $i\in[J+1,m]$. Hence, we have

[TABLE]

Note that the first $J$ CNs cover $(n-\theta)$ VNs, that is to say $\sum_{l=1}^{J}|\mathcal{Y}_{l}|=n-\theta$. Thus, from (21), we have

[TABLE]

Considering (20) to (23) and noting that $\max\limits_{l\in[1,J-\delta_{i}]}|\mathcal{Y}_{l}|=\theta_{i}$, $\delta_{i}\leq J-1$, and \theta_{i}\leq\theta\leq d-2<\Big{\lceil}\frac{k}{J}\Big{\rceil}+1, we have the following chain of inequalities

[TABLE]

In other words,

[TABLE]

By replacing (24) in (19), we have:

[TABLE]

where $\sum_{i=1}^{J}|\mathcal{Y}_{i}|=n-\theta$. Hence, for $\theta\in[1,d-2]$

[TABLE]

By Lemma 4, we have

[TABLE]

Combining (18) and (27) concludes the proof.