Optimal repair of Reed-Solomon codes: Achieving the cut-set bound

TL;DR

This paper constructs Reed-Solomon codes that achieve the optimal repair bandwidth bound in distributed storage, solving a long-standing open problem by matching the cut-set bound with super-exponential sub-packetization.

Contribution

The authors construct RS codes over large fields that meet the cut-set bound for repair bandwidth, and establish a lower bound on sub-packetization necessary for such optimal repair.

Findings

RS codes constructed over fields with super-exponential size meet the cut-set bound.

Lower bound on sub-packetization shows super-exponential growth is necessary.

The work resolves the open problem of achieving optimal repair bandwidth with scalar MDS codes.

Abstract

Coding for distributed storage gives rise to a new set of problems in coding theory related to the need of reducing inter-node communication in the system. A large number of recent papers addressed the problem of optimizing the total amount of information downloaded for repair of a single failed node (the repair bandwidth) by accessing information on $d$ {\em helper nodes}, where $k\le d\le n-1.$ By the so-called cut-set bound (Dimakis et al., 2010), the repair bandwidth of an $(n,k=n-r)$ MDS code using $d$ helper nodes is at least $dl/(d+1-k),$ where $l$ is the size of the node. Also, a number of known constructions of MDS array codes meet this bound with equality. In a related but separate line of work, Guruswami and Wootters (2016) studied repair of Reed-Solomon (RS) codes, showing that these codes can be repaired using a smaller bandwidth than under the trivial approach. At the same time, their work as well as follow-up papers stopped short of constructing RS codes (or any scalar MDS codes) that meet the cut-set bound with equality, which has been an open problem in coding theory. In this work we present a solution to this problem, constructing RS codes of length $n$ over the field $q^l, l=\exp((1+o(1))n\log n)$ that meet the cut-set bound. We also prove an almost matching lower bound on $l$, showing that the super-exponential scaling is both necessary and sufficient for achieving the cut-set bound using linear repair schemes. More precisely, we prove that for scalar MDS codes (including the RS codes) to meet this bound, the sub-packetization $l$ must satisfy $l \ge \exp((1+o(1)) k\log k).$

Tables

Table 1. TABLE I: Tradeoff between repair bandwidth and sub-packetization

Code construction	Repair bandwidth	sub-packetization	achieving cut-set bound
Array codes
$\left(n,k=n-r,n-1,l\right)$ MSR array codes for $2k\le \left(n+1\right)$, [9]	$\frac{\left(n-1\right)l}{r}$	$l=r$	Yes
$(n,k,n-1,l)$ MSR array codes (a modification of [18])	$\frac{\left(n-1\right)l}{r}$	$l=r^{\lceil n/\left(r+1\right)\rceil }$	Yes
$(n,k,n-1,l)$ MSR array codes [5]	$\frac{\left(n-1\right)l}{r}$	$l=r^{\lceil n/r\rceil }$	Yes
$(n,k)$ MDS array codes with design parameter $t\ge 1$ [19]	$\left(1+\frac{1}{t}\right)\frac{\left(n-1\right)l}{r}$	$l=r^t$	No
Scalar codes
$(n,k)$ RS code [14]		$l=r^n$	No
$(n,k)$ RS code [13]	$n-1$	$l={\log }_{n/r}n$	No
$(n,k)$ RS code [15]	$\left(n-1\right)l\left(1-{\log }_{n}r\right)$	${\log }_{q}n$	No
$(n,k)$ RS code (this paper)	$\frac{\left(n-1\right)l}{r}$	$l\approx n^n$	Yes

Equations

$$\underset{\begin{array}{c}i\in [n]\\ \mathcal{R}\subseteq [n]\setminus \{i\},|\mathcal{R}|=d\end{array}}{\max }N(\mathcal{C},i,\mathcal{R}).$$

$$N(\mathcal{C},i,\mathcal{R})\ge \frac{dl}{d+1-k}.$$

$$\mathcal{C}^{\perp}=\big{\{}x=(x_{1},\dots,x_{n})\in E^{n}\big{|}\sum_{i=1}^{n}x_{i}c_{i}=0\quad\forall c=(c_{1},\dots c_{n})\in\mathcal{C}\big{\}}.$$

$${\dim }_F(\{x_i:x\in {\mathcal{P}}_i\})=l,$$

$$b\ge \sum _{j\in [n]\setminus \{i\}}{\dim }_F(\{x_j:x\in {\mathcal{P}}_i\})$$

$$l=s\biggl{(}\prod_{\begin{subarray}{c}i=1\\[1.4457pt] p_{i}\equiv 1\text{ mod }s\end{subarray}}^{n}p_{i}\biggr{)},$$

$$e^{(1+o(1))k\log k}\le l\le e^{(1+o(1))n\log n}.$$

$$\{(v_{1}f({\omega }_1),\dots ,v_{n}f({\omega }_n))\in F^n:f\in F[x],\mathrm{deg}f\le k-1\}$$

$$({\text{RS}}_E(n,k,\Omega ){)}^{\perp }={\text{GRS}}_E(n,n-k,\Omega ,v)$$

$${\mathrm{tr}}_{E/F}(x):=x+x^{|F|}+x^{|F{|}^2}+\cdots +x^{|F{|}^{t-1}}.$$

$$F_i={\mathbb{F}}_q({\alpha }_j:j\ne i),$$

$$d_i=p_i+k-1.$$

$$h(x)=\prod _{j\in [n]\setminus ({\mathcal{R}}_i\cup \{i\})}(x-{\alpha }_j).$$

$$(v_1{\alpha }_1^{s}h({\alpha }_1),\dots ,v_n{\alpha }_n^{s}h({\alpha }_n))\in {\mathcal{C}}^{\perp },$$

$$\sum _{j=1}^n{v}_{j}h({\alpha }_j){\alpha }_j^s{c}_j=0,s=0,\dots ,p_i-1.$$

$$\sum _{j=1}^n{\mathrm{tr}}_i(v_{j}h({\alpha }_j){\alpha }_j^s{c}_j)=0,s=0,\dots ,p_i-1.$$

$${\mathrm{tr}}_i(v_{i}h({\alpha }_i){\alpha }_i^s{c}_i)$$

$$=-\sum _{j\in {\mathcal{R}}_i}{\mathrm{tr}}_i(v_{j}h({\alpha }_j){\alpha }_j^s{c}_j)$$

$$=-\sum _{j\in {\mathcal{R}}_i}{\alpha }_j^s{\mathrm{tr}}_i(v_{j}h({\alpha }_j)c_j),s=0,\dots ,p_i-1,$$

$${\mathrm{tr}}_i(v_{j}h({\alpha }_j)c_j),j\in {\mathcal{R}}_i.$$

$$\begin{array}[]{ll}E\to F_{i}^{p_{i}}\\ \gamma\mapsto\big{(}\operatorname{tr}_{i}\big{(}v_{i}h(\alpha_{i})\gamma\big{)},\operatorname{tr}_{i}\big{(}v_{i}h(\alpha_{i})\alpha_{i}\gamma\big{)},\dots,\operatorname{tr}_{i}\big{(}v_{i}h(\alpha_{i})\alpha_{i}^{p_{i}-1}\gamma\big{)}\big{)}.\end{array}$$

$$\operatorname{tr}_{i}(v_{i}h(\alpha_{i})\alpha_{i}^{s}\theta_{j})=\left\{\begin{array}[]{cc}0,&\text{~{}if~{}}s\neq j\\ 1,&\text{~{}if~{}}s=j\end{array}\right.\text{for all~{}}s,j\in\{0,1,\dots,p_{i}-1\}.$$

$$c_i=\sum _{s=0}^{p_i-1}{\mathrm{tr}}_i(v_{i}h({\alpha }_i){\alpha }_i^s{c}_i){\theta }_s.$$

$${\alpha }_1={\alpha }^{\frac{5^{30}-1}{5^2-1}},{\alpha }_2={\alpha }^{\frac{5^{30}-1}{5^3-1}},{\alpha }_3={\alpha }^{\frac{5^{30}-1}{5^5-1}},{\alpha }_4=0,{\alpha }_5=1,{\alpha }_6=2,{\alpha }_7=3,{\alpha }_8=4.$$

$${\mathbb{F}}_5({\alpha }_i:i\in A)={\mathbb{F}}_{m_{{}_A}},\text{ where }{m}_{{}_A}=5^{({\prod }_{i\in A}{p}_i)}.$$

$${\mathrm{tr}}_{E/{\mathbb{F}}_{5^6}}({\alpha }_3^s{v}_3{c}_3)=-\sum _{j\ne 3}{\mathrm{tr}}_{E/{\mathbb{F}}_{5^6}}({\alpha }_j^s{v}_j{c}_j)=-\sum _{j\ne 3}{\alpha }_j^s{\mathrm{tr}}_{E/{\mathbb{F}}_{5^6}}(v_j{c}_j),s=0,\dots ,4.$$

$$p_i\equiv 1\text{mod}s\text{ for all }i=1,2,\dots ,n.$$

$$\mathbb{F}:={\mathbb{F}}_p({\alpha }_1,\dots ,{\alpha }_n).$$

$${\dim }_{F_i}{S}_i=p_i,S_i+S_i{\alpha }_i+\cdots +S_i{\alpha }_i^{s-1}=\mathbb{K},$$

$$h(x)=\prod _{j\in [n]\setminus (\mathcal{R}\cup \{i\})}(x-{\alpha }_j).$$

$$(v_1{\alpha }_1^{t}h({\alpha }_1),\dots ,v_n{\alpha }_n^{t}h({\alpha }_n))\in {\mathcal{C}}^{\perp }.$$

Full text

By the so-called cut-set bound (Dimakis et al., 2010), the repair bandwidth of an $(n,k=n-r)$ MDS code using $d$ helper nodes is at least $dl/(d+1-k),$ where $l$ is the size of the node. Also, a number of known constructions of MDS array codes meet this bound with equality. In a related but separate line of work, Guruswami and Wootters (2016) studied repair of Reed-Solomon (RS) codes, showing that these codes can be repaired using a smaller bandwidth than under the trivial approach. At the same time, their work as well as follow-up papers stopped short of constructing RS codes (or any scalar MDS codes) that meet the cut-set bound with equality, which has been an open problem in coding theory.

In this work we present a solution to this problem, constructing RS codes of length $n$ over the field $q^{l},l=\exp((1+o(1))n\log n)$ that meet the cut-set bound. We also prove an almost matching lower bound on $l$, showing that the super-exponential scaling is both necessary and sufficient for achieving the cut-set bound using linear repair schemes. More precisely, we prove that for scalar MDS codes (including the RS codes) to meet this bound, the sub-packetization $l$ must satisfy $l\geq\exp((1+o(1))k\log k).$

††footnotetext:

  I. Tamo is with Department of EE-Systems, Tel Aviv University, Tel Aviv, Israel. Email: zactamo@gmail.com. His research is supported by ISF grant no. 1030/15 and the NSF-BSF grant no. 2015814. M. Ye and A. Barg are with Dept. of ECE and ISR, University of Maryland, College Park, MD 20742. Emails: yeemmi@gmail.com and abarg@umd.edu. Their research is supported by NSF grants CCF1422955 and CCF1618603.

I Introduction

I-A Minimum Storage Regenerating codes and optimal repair bandwidth

The amount of information produced has grown exponentially over the last decade, and large-scale storage systems are widely used to store the data. The problem that we consider is motivated by applications of codes in distributed storage wherein the data is written on a large number of physical storage nodes. Failure of an individual node renders a portion of the data inaccessible, and erasure correcting codes are used to increase the reliability of the system. The repair task performed by the system relies on communication between individual nodes, and introduces new challenges in the code design. In particular, a new parameter that has a bearing on the overall efficiency of the system is the amount of data sent between the nodes in the process of repair.

To protect the information, we divide the original file into $k$ information blocks and view each block as a single element of a finite field $F$ or a vector over $F$. We encode the data by adding $r=n-k$ parity blocks (field symbols or vectors) and distributing the resulting $n$ blocks across $n$ storage nodes. In this paper we deal only with linear codes, so the parity blocks are formed as linear combinations of the information blocks over $F.$ We use the notation $(n,k)$ to refer to the length and dimension of a linear code. A well-known class of linear $(n,k)$ Maximum Distance Separable (MDS) codes, studied in this paper, has the favorable property that the original file can be recovered from the content stored on any $k$ nodes, which provides the optimal tradeoff between failure tolerance and storage overhead.

Before proceeding further, we make a brief remark on the terminology used in the literature devoted to erasure correcting codes for distributed storage. The coordinates of the codeword are assumed to be stored on different nodes, and by extension are themselves referred to as nodes. In practice, single node failure is the most common scenario [1, Section 6.6], so we will be interested in the problem of designing codes that efficiently correct (repair) a single erasure (failed node). We assume that the data is encoded with a code $\mathcal{C}$ over a finite field $F$ wherein each coordinate of the codeword is either an element of $F$ or an $l$-dimensional vector over $F$, where $l>1.$ The latter construction, termed array codes turns out to be better suited to the needs of the repair problem, as will be apparent in the later part of this section. To repair a failed node, the system needs to download the contents from some other nodes (helper nodes) of the codeword to the processor, and the total amount of the downloaded data is called the repair bandwidth. Coding solutions that support efficient repair are called regenerating codes, and they have been a focal point of current research in coding theory following their introduction in [2].

One traditional solution to recover a single node failure in an MDS-coded system is to download the content stored on any $k$ nodes. The MDS property guarantees that we can recover the whole file, so we can also recover any single node failure. However, this method is far from efficient in the sense that the repair bandwidth that it requires is much larger than is needed for the repair of a single node. Indeed, by a rather counter-intuitive result of [2] it is possible to save on the repair bandwidth by contacting $d>k$ helper nodes, and the maximum savings are attained when $d$ is the largest possible value, namely $d=n-1$.

More specifically, suppose that an $(n,k)$ MDS-coded system attempts to repair a failed node by connecting to $d$ helper nodes. In this case, as shown in [2], the total amount of information that needs to be downloaded to complete the repair task is at least $dl/(d+1-k),$ where $l$ is the size of each node. This lower bound on the repair bandwidth is called the cut-set bound since it is obtained from the cut-set bound in network information theory [3]. Given $k<d\leq n-1,$ an $(n,k)$ MDS code achieving the cut-set bound for the repair of any single failed node from any $d$ helper nodes is called an $(n,k)$ minimum storage regenerating (MSR) code with repair degree $d$ [2].

The definition of MSR codes, given above in an informal way, will be formalized for a particular subclass of codes known as MDS array codes. An $(n,k)$ MDS array code $\mathcal{C}$ with sub-packetization $l$ over a finite field $F$ is formed of $k$ information nodes and $r=n-k$ parity nodes, where every node is a column vector of length $l$ over $F$ (so $\dim_{F}(\mathcal{C})=kl$). The MDS property requires that any $k$ nodes of $\mathcal{C}$ suffice to recover the remaining $r$ nodes of the codeword. Array codes are also called vector codes, while code families more common to coding theory (such as Reed-Solomon (RS) codes and others) are called scalar codes in the literature. Clearly, scalar codes correspond to the case $l=1$ of the above definition.

Definition 1** (Repair bandwidth).**

Let $\mathcal{C}$ be an $(n,k)$ MDS array code with sub-packetization $l$ over a finite field $F$. For $i\in\{1,\dots,n\}$ and $\mathcal{R}\subseteq[n]\setminus\{i\}$ with cardinality $|\mathcal{R}|\geq k$, define $N(\mathcal{C},i,\mathcal{R})$ as the smallest number of symbols of $F$ one needs to download from the helper nodes $\{c_{j}:j\in\mathcal{R}\}$ in order to repair the failed node $c_{i}$. The repair bandwidth of the code $\mathcal{C}$ with $d$ helper nodes equals

[TABLE]

We note that the symbols downloaded to repair the node $c_{i}$ can be some functions of the contents of the helper nodes $\{c_{j},j\in\mathcal{R}\}.$

Definition 2** (Cut-set bound [2]).**

Let $\mathcal{C}$ be an $(n,k)$ MDS array code with sub-packetization $l$ and let $k\leq d\leq n-1$. For any $i\in[n]$ and any subset $\mathcal{R}\subseteq[n]\setminus\{i\}$ of size $d$ we have the following inequality:

[TABLE]

An $(n,k)$ MDS array code with sub-packetization $l$ achieving the cut-set bound (1) for the repair of any single failed node from any $d$ helper nodes is called an $(n,k,d,l)$ MSR array code.

Several constructions of MSR codes are available in the literature: See [4, 5, 6, 7, 8] for the high-rate regime where $k>n/2$, and [9] for the low-rate regime where $k\leq n/2$. Recently the concept of repair bandwidth was extended in [10] to the problem of correcting errors; this paper also presented explicit code constructions that support error correction under the minimum possible amount of information downloaded during the decoding process.

Due to the limited storage capacity of each node, we would like the sub-packetization $l$ to be as small as possible. However, it is shown in [11] that for an $(n,k,d=n-1,l)$ MSR array code, $l\geq\exp({\sqrt{k/(2r-1)}})$ (i.e., $l$ is exponential in $n$ for fixed $r$ and growing $n$).

I-B Repair schemes for scalar linear MDS codes

While there has been much research into constructions and properties of MSR codes specifically designed for the repair task, it is also of interest to study the repair bandwidth of general families of MDS codes, for instance, RS codes. In [12], Shanmugam et al. proposed a framework for studying the repair bandwidth of a scalar linear $(n,k)$ MDS code $\mathcal{C}$ over some finite field $E$ (called symbol field below). The idea of [12] is to “vectorize” the code construction by considering $\mathcal{C}$ as an array code over some subfield $F$ of $E$. This approach provides a bridge between RS codes and MDS array codes, wherein the extension degree $l:=[E:F]$ can be viewed as the value of sub-packetization. The code $\mathcal{C}$ is viewed as an $(n,k)$ MDS array code with sub-packetization $l$, and the repair bandwidth is defined exactly in the same way as above. The cut-set bound (1) and the definition of MSR codes also apply to this setup.

In this paper we study repair of RS codes, focusing on linear repair schemes, i.e., we assume that the repair operations are linear over the field $F.$ Guruswami and Wootters [13] gave a characterization for linear repair schemes of scalar linear MDS codes based on the framework in [12]. We will use this characterization to prove one of our main results, namely, a lower bound on the sub-packetization, so we recall it below. Let us start with the definition of the dual code.

Definition 3** (Dual code).**

The dual code of a linear code $\mathcal{C}\subseteq E^{n}$ is the linear subspace of $E^{n}$ defined by

[TABLE]

In the next theorem $E$ is the degree-$l$ extension of the field $F$. Viewing $E$ as an $l$-dimensional vector space over $F$, we use the notation $\dim_{F}(a_{1},a_{2},\dots,a_{t})$ to refer to the dimension of the subspace spanned by the set $\{a_{1},a_{2},\dots,a_{t}\}\subset E$ over $F$.

We will need a result from [13] which we state in the form that is suited to our needs.

Theorem 1** ([13]).**

Let $\mathcal{C}\subseteq E^{n}$ be a scalar linear MDS code of length $n$. Let $F$ be a subfield of $E$ such that $[E:F]=l.$ For a given $i\in\{1,\dots,n\}$ the following statements are equivalent.

1. (1)

There is a linear repair scheme of the node $c_{i}$ over $F$ such that the repair bandwidth $N(\mathcal{C},i,[n]\setminus\{i\})\leq b$. 2. (2)

There is a subset of codewords $\mathcal{P}_{i}\subseteq\mathcal{C}^{\perp}$ with size $|\mathcal{P}_{i}|=l$ such that

[TABLE]

and

[TABLE]

In addition to this general linear repair scheme for scalar linear MDS codes, the authors of [13] also presented a specific repair scheme for a family of RS codes and further proved that (in some cases) the repair bandwidth of RS codes using this scheme is the smallest possible among all linear repair schemes and all scalar linear MDS codes with the same parameters. At the same time, the approach of [13] has some limitations. Namely, their repair scheme applies only for small sub-packetization $l=\log_{n/r}n$, and the optimality claim only holds for this specific sub-packetization value. At the same time, in order to achieve the cut-set bound, $l$ needs to be exponentially large in $n$ for a fixed value of $r$ [11], so the repair bandwidth of this scheme is rather far from the bound. Subsequently, Ye and Barg [14] used the general linear repair scheme in [13] to construct an explicit family of RS codes with asymptotically optimal repair bandwidth: the ratio between the actual repair bandwidth of the codes and the cut-set bound approaches $1$ as the code length $n$ goes to infinity.

In [13], there is one more restriction on the parameters of the RS codes, namely they achieve the smallest possible repair bandwidth only if the number of parities is of the form $r=q^{s},(l-s)|l.$ In [15], Dau and Milenkovic generalized the scheme in [13] and extended their results to all values of $s=1,\dots,l-1$. The repair bandwidth attained in [15] is $(n-1)(l-s)$ symbols of $F$ for $r\geq q^{s}$, and is the smallest possible whenever $r$ is a power of $q.$ In [16], Dau et al. extended the results of [13] to repair of multiple erasures.

To summarize the earlier work, constructions of RS codes (or any scalar MDS codes) that meet the cut-set bound have as yet been unknown, so the existence question of such codes has been an open problem. In this paper, we resolve this problem in the affirmative, presenting such a construction. We also prove a lower bound on the sub-packetization of scalar linear MDS codes that attain the cut-set bound with a linear repair scheme, showing that there is a penalty for the scalar case compared to MDS array codes.

I-C Our Results

1. (1)

Explicit constructions of RS codes achieving the cut-set bound: Given any $n,k$ and $d,k\leq d\leq n-1$, we construct an $(n,k)$ RS code over the field $E={\mathbb{F}}_{q^{l}}$ that achieves the cut-set bound (1) when repairing any single failed node from any $d$ helper nodes. As above, we view RS codes over $E$ as vector codes over the subfield $F={\mathbb{F}}_{q}$. The main novelty in our construction is the choice of the evaluation points for the code in such a way that the degrees of the evaluation points over $F$ are distinct primes. As a result, the symbol field is an extension field of $F$ with degree no smaller than the product of these distinct primes. For the actual repair we rely on the linear scheme proposed in [13] (this is essentially the only possible linear repair approach).

The value of sub-packetization $l$ of our construction equals $s$ times the product of the first $n$ distinct primes in an arithmetic progression,

[TABLE]

where $s:=d+1-k.$ This product is a well-studied function in number theory, related to a classical arithmetic function $\psi(n,s,a)$ (which is essentially the sum of logarithms of the primes). The prime number theorem in arithmetic progressions (for instance, [17, p.121]) yields asymptotic estimates for $l$. In particular, for fixed $s$ and large $n$, we have $l=e^{(1+o(1))n\log n}.$

In contrast, for the case $d=n-1$ (i.e., $s=r=n-k$), there exist MSR array codes that attain sub-packetization $l=r^{\lceil n/(r+1)\rceil}$ [18], which is the smallest known value among MSR codes111 The construction of [18] achieves the cut-set bound only for repair of systematic nodes, and gives $l=r^{\lceil k/(r+1)\rceil}$. Using the approach of [4], it is possible to modify the construction of [18] and to obtain an MSR code with $l=r^{\lceil n/(r+1)\rceil}$.. So although this distinct prime structure allows us to achieve the cut-set bound, it makes us pay a penalty on the sub-packetization. 2. (2)

A lower bound on the sub-packetization of scalar MDS codes achieving the cut-set bound: Surprisingly, we also show that the distinct prime structure discussed above is necessary for any scalar linear MDS code (not just the RS codes) to achieve the cut-set bound under linear repair. Namely, given $d$ such that $k+1\leq d\leq n-1,$ we prove that for any $(n,k)$ scalar linear MSR code with repair degree $d,$ the sub-packetization $l$ is bounded below by $l\geq\prod_{i=1}^{k-1}p_{i}$, where $p_{i}$ is the $i$-th smallest prime. By the Prime Number Theorem [17], we obtain the lower asymptotic bound on $l$ of the form $l\geq e^{(1+o(1))k\log k}.$ 3. (3)

Main result: In summary, we obtain the following results for the smallest possible sub-packetization of scalar linear MDS codes, including the RS codes, whose repair bandwidth achieves the cut-set bound.

Theorem 2**.**

Let $\mathcal{C}$ be an $(n,k=n-r)$ scalar linear MDS code over the field $E={\mathbb{F}}_{q^{l}},$ and let $d$ be an integer satisfying $k+1\leq d\leq n-1.$ Suppose that for any single failed node of $\mathcal{C}$ and any $d$ helper nodes there is a linear repair scheme over ${\mathbb{F}}_{q}$ that uses the bandwidth $dl/(d+1-k)$ symbols of ${\mathbb{F}}_{q}$, i.e., it achieves the cut-set bound (1). For a fixed $s=d+1-k$ and $n,k\to\infty$ the following bounds on the smallest possible sub-packetization hold true:

[TABLE]

For large $s$, we have $l\leq s\prod\limits_{\begin{subarray}{c}i:p_{i}\equiv 1\,\text{\rm mod}\,s\end{subarray}}^{n}p_{i},$ where the product goes over the first $n$ distinct primes in the arithmetic progression.

Remark 1**.**

The bound on $l$ can be made more explicit even for large $s$, and the answer depends on whether we accept the Generalized Riemann Hypothesis (if yes, we can still claim the bound $l\leq\exp((1+o(1))n\log n)$). 4. (4)

Discussion: Array codes and scalar codes The lower bound in (2) is much larger than the sub-packetization of many known MSR array code constructions. To make the comparison between the repair parameters of scalar codes and array codes more clear, we summarize the tradeoff between the repair bandwidth and the sub-packetization of some known MDS code constructions in the following table. We only list the papers considering the repair of a single node from all the remaining $n-1$ helper nodes. Moreover, in the table we limit ourselves to explicit code constructions, and do not list multiple existence results that appeared in recent years.

As discussed earlier, the constructions of [13, 15] have optimal repair bandwidth among all the RS codes with the same sub-packetization value as in these papers222Expressing the sub-packetization of the construction in [15] via $n$ and $r$ is difficult. The precise form of the result in [15] is as follows: for every $s<l$ and $r\geq q^{s},$ the authors construct repair schemes of RS codes of length $n=q^{l}$ with repair bandwidth $(n-1)(l-s).$ Moreover, if $r=q^{s},$ then the schemes proposed in [15] achieve the smallest possible repair bandwidth for codes with these parameters.. At the same time, these values are too small for the constructions of [13, 15] to achieve the cut-set bound. From the first three rows of the table one can clearly see that the achievable sub-packetization values for MSR array codes are much smaller than the lower bound for scalar linear MSR codes derived in this paper. This is to be expected since for array codes we only require the code to be linear over the “repair field,” i.e., $F$, and not the symbol field $E$ as in the case of scalar codes.

I-D Organization of the paper

In Sec. II, we present a simple construction of RS codes that achieve the cut-set bound for some of the nodes. This construction is inferior to the more involved construction of Sec. III, but simple to follow, and already contains some of the main ideas of the later part, so we include it as a warm-up for the later results. In Sec. III, we present our main construction of RS codes that achieve the cut-set bound for the repair of any single node, proving the upper estimate in (2). In Sec. IV, we prove the lower bound on the sub-packetization of scalar linear MSR codes, finishing the proof of (2).

II A simple construction

In this section we present a simple construction of RS codes that achieve the cut-set bound for the repair of certain nodes. We note that any $(n,k)$ MDS code trivially allows repair that achieves the cut-set bound for $d=k$. We say that a node in an MDS code has a nontrivial optimal repair scheme if for a given $d>k$ it is possible to repair this node from any $d$ helper nodes with repair bandwidth achieving the cut-set bound. The code family presented in this section is different from standard MSR codes in the sense that although the repair bandwidth of our construction achieves the cut-set bound, the number of helper nodes depends on the node being repaired.

Let us first recall the definition of (generalized) Reed-Solomon codes.

Definition 4**.**

A Generalized Reed-Solomon code $\text{\rm GRS}_{F}(n,k,\Omega,v)\subseteq F^{n}$ of dimension $k$ over $F$ with evaluation points $\Omega=\{\omega_{1},\omega_{2},\dots,\omega_{n}\}\subseteq F$ is the set of vectors

[TABLE]

where $v=(v_{1},\dots,v_{n})\in(F^{\ast})^{n}$ are some nonzero elements. If $v=(1,\dots,1),$ then the GRS code is called a Reed-Solomon code and is denoted as $\text{\rm RS}_{F}(n,k,\Omega)$.

It is well known [20, p.304] that

[TABLE]

where $v_{i}=\prod_{j\neq i}(\omega_{i}-\omega_{j})^{-1},i=1,\dots,n$. (The dual of an RS code is a GRS code.)

Denote by $\pi(t)$ the number of primes less than or equal to $t$. Let $F$ be a finite field and let $E$ be the extension of $F$ of degree $t.$ The trace function $\operatorname{tr}_{E/F}:E\to F$ is defined by

[TABLE]

In the next theorem we construct a special subfamily of RS codes. Our construction enables nontrivial repair of $\pi(r)$ nodes, which without loss of generality we take to be nodes $1,2,\dots,\pi(r)$. Let $d_{i},i=1,2,\dots,\pi(r)$ be the number of helper nodes used to repair the $i$-th node. We will take $d_{i}=p_{i}+k-1$, where $p_{i}$ is the $i$-th smallest prime number. The repair scheme presented below supports repair of node $i$ by connecting to any $d_{i}$ helper nodes and downloading a $\frac{1}{p_{i}}$-th proportion of information stored at each of these nodes. Since $p_{i}=d_{i}-k+1,$ this justifies the claim of achieving the cut-set bound for repair of a single node.

Theorem 3**.**

Let $n\geq k$ be two positive integers, and let $r=n-k.$ There exists an $(n,k)$ RS code over a field $E$ such that $\pi(r)$ of its coordinates admit nontrivial optimal repair schemes.

Proof:

Let $m:=\pi(r)$ and let $q\geq n-m$ be a prime power. Let $E$ be the \big{(}\prod_{i=1}^{m}p_{i}\big{)}-th degree extension of the finite field $\mathbb{F}_{q}$.

Let $\alpha_{i},i=1,\dots,m$ be an element of order $p_{i}$ over $\mathbb{F}_{q},$ so that $\mathbb{F}_{q^{p_{i}}}=\mathbb{F}_{q}(\alpha_{i}),$ where $\mathbb{F}_{q}(\alpha_{i})$ denotes the field obtained by adjoining $\alpha_{i}$ to $\mathbb{F}_{q}.$ It is clear that $E=\mathbb{F}_{q}(\alpha_{1},\dots,\alpha_{m})$. Define $m$ subfields $F_{i}$ of $E$ by setting

[TABLE]

so that $E=F_{i}(\alpha_{i})$ and $[E:F_{i}]=p_{i}$, $i=1,\dots,m.$ Let $\alpha_{m+1},\dots,\alpha_{n}\in\mathbb{F}_{q}$ be arbitrary $n-m$ distinct elements of the field, and let $\Omega=\{\alpha_{1},\alpha_{2},\dots,\alpha_{n}\}$.

Let $\mathcal{C}=\text{\rm RS}_{E}(n,k,\Omega)$ be the RS code of dimension $k$ with evaluation points $\Omega$ and let $\mathcal{C}^{\bot}$ be its dual code. We claim that for $i=1,2,\dots,m,$ the $i$-th coordinate (node) of $\mathcal{C}$ can be optimally repaired from any $d_{i}$ helper nodes, where

[TABLE]

Let $i\in\{1,2,\dots,m\}$ and let us show how to repair the $i$th node. Choose a subset of helper nodes $\mathcal{R}_{i}\subseteq[n]\backslash\{i\},|\mathcal{R}_{i}|=d_{i},$ and note that since $p_{i}\leq r,$ we have $d_{i}\leq n-1$. Let $h(x)$ be the annihilator polynomial of the set $\{\alpha_{j}:j\in[n]\setminus(\mathcal{R}_{i}\cup\{i\})\}$, i.e.,

[TABLE]

Since $\deg(h(x))=n-k-p_{i},$ we have $\deg(x^{s}h(x))<r$ for all $s=0,1,\dots,p_{i}-1.$ As a result, for all $s=0,\dots,p_{i}-1$, the vector

[TABLE]

cf. (3). Let $c=(c_{1},\dots,c_{n})\in\mathcal{C}$ be a codeword. By (5) we have

[TABLE]

Let $\operatorname{tr}_{i}:=\operatorname{tr}_{E/F_{i}}$ denote the trace from $E$ to $F_{i}.$ We have

[TABLE]

Equivalently, we can write

[TABLE]

where the second equality follows from (4) and the third follows because $\alpha_{j}\in F_{i}$ for all $j\neq i$ and $\operatorname{tr}_{i}$ is an $F_{i}$-linear map.

The information used to recover the value $c_{i}$ (to repair the $i$th node) is comprised of the following $d_{i}$ elements of $F_{i}:$

[TABLE]

Let us show that these elements indeed suffice. First, by (6), given these elements, we can calculate the values of $\operatorname{tr}_{i}(v_{i}h(\alpha_{i})\alpha_{i}^{s}c_{i})$ for all $s=0,\dots,p_{i}-1.$ The mapping

[TABLE]

is in fact a bijection, which can be realized as follows. Since the set $\{1,\alpha_{i},\dots,\alpha_{i}^{p_{i}-1}\}$ forms a basis of $E$ over $F_{i}$ and $v_{i}h(\alpha_{i})\neq 0,$ the set $\{v_{i}h(\alpha_{i}),v_{i}h(\alpha_{i})\alpha_{i},\dots,v_{i}h(\alpha_{i})\alpha_{i}^{p_{i}-1}\}$ also forms a basis. Let $\{\theta_{0},\theta_{1},\dots,\theta_{p_{i}-1}\}$ be the dual basis of $\{v_{i}h(\alpha_{i}),v_{i}h(\alpha_{i})\alpha_{i},\dots,v_{i}h(\alpha_{i})\alpha_{i}^{p_{i}-1}\},$ i.e.,

[TABLE]

The value $c_{i}$ can now be found as follows:

[TABLE]

(this is the essence of the repair scheme proposed in [13]).

The presented arguments constitute a linear repair scheme of the node $c_{i},i=1,\dots m$ over $F_{i}.$ The information downloaded from each of the helper nodes consists of one element of $F_{i},$ or, in other words, the $(1/p_{i})$th proportion of the contents of each node. This shows that node $i$ admits nontrivial optimal repair. The proof is thereby complete. ∎

Example 1**.**

Take $q=5,$ $k=3,r=5$. We have $\pi(r)=3$ and $p_{1}=2,p_{2}=3,p_{3}=5.$ Let us construct an $(8,3)$ RS code over the field $E=\mathbb{F}_{5^{30}}$, where the first $3$ nodes admit nontrivial optimal repair schemes. Let $\alpha$ be a primitive element of $E$. Choose the set $\Omega=\{\alpha_{1},\dots,\alpha_{8}\}$ as follows:

[TABLE]

The number of helper nodes for the first $3$ nodes is $(d_{1},d_{2},d_{3})=(4,5,7)$. It is easy to verify that for any subset $A\subseteq\{1,2,3\}$

[TABLE]

The code $\mathcal{C}$ constructed in the above proof is given by $\mathcal{C}=\text{\rm RS}_{E}(8,3,\Omega).$ Let us address the task of repairing $c_{3}$ from all the remaining $7$ helper nodes with repair bandwidth achieving the cut-set bound. Let $\mathcal{C}^{\bot}=\text{\rm GRS}_{E}(8,5,\Omega,v)$, where $v=(v_{1},\dots,v_{8})\in(E^{\ast})^{8}$. We download the value $\operatorname{tr}_{E/\mathbb{F}_{5^{6}}}(v_{j}c_{j})$ from each helper node $c_{j},j\neq 3$. Since $[E:\mathbb{F}_{5^{6}}]=p_{3},$ this amounts to downloading exactly a $1/p_{3}=(1/5)$-th fraction of the information stored at each helper node, which achieves the cut-set bound. The value of $c_{3}$ can be found from the downloaded information using the following $5$ equations:

[TABLE]

Indeed, the downloaded symbols suffice to recover the vector $(\operatorname{tr}_{E/\mathbb{F}_{5^{6}}}(\alpha_{3}^{s}v_{3}c_{3}),{s=0,\dots,4})$, and therefore also suffice to repair the symbol $c_{3}$.

III A family of RS codes achieving the cut-set bound

In this section we develop the ideas discussed above and construct RS codes achieving the cut-set bound with nontrivial optimal repair of all nodes. More precisely, given any positive integers $k<d\leq n-1$, we explicitly construct an $(n,k)$ RS code $\mathcal{C}$ achieving the cut-set bound for the repair of any single node from any $d$ helper nodes. In other words, $\mathcal{C}$ is an $(n,k)$ MSR code with repair degree $d$.

Let $\mathbb{F}_{p}$ be a finite field of prime order (for simplicity we can take $p=2$). Denote $s:=d-k+1$ and let $p_{1},\dots,p_{n}$ be $n$ distinct primes such that

[TABLE]

According to Dirichlet’s theorem, there are infinitely many such primes. For $i=1,\dots,n$, let $\alpha_{i}$ be an element of degree $p_{i}$ over $\mathbb{F}_{p}$, i.e., $[\mathbb{F}_{p}(\alpha_{i}):\mathbb{F}_{p}]=p_{i}$, and define

[TABLE]

Note that for any subset of indices $A\subseteq[n]$, the field $\mathbb{F}_{p}(\{\alpha_{i}:i\in A\})$ is an extension of $\mathbb{F}_{p}$ of degree $\prod_{i\in A}p_{i},$ and in particular, $\mathbb{F}$ has degree $\prod_{i=1}^{n}p_{i}$ over $\mathbb{F}_{p}$. Next, we define $n$ distinct subfields $F_{i}$ of the field $\mathbb{F}$ and one extension field $\mathbb{K}$ of $\mathbb{F}$.

For $i=1,\dots,n$, define $F_{i}=\mathbb{F}_{p}(\{\alpha_{j}:j\neq i\})$. Note that $\mathbb{F}=F_{i}(\alpha_{i})$ and $[\mathbb{F}:F_{i}]=p_{i}$. 2. 2.

The field $\mathbb{K}$ is defined to be the degree-$s$ extension of the field $\mathbb{F}$, i.e. there exists an element $\beta\in\mathbb{K}$ of degree $s$ over $\mathbb{F}$ such that $\mathbb{K}=\mathbb{F}(\beta).$ We also have $[\mathbb{K}:F_{i}]=sp_{i}$ for all $i$.

We are ready to construct a family of RS codes that can be optimally repaired for each node. The set $\alpha_{1},\dots,\alpha_{n}$ serves as the set of evaluation points of the code.

The following theorem is the main result of this section.

Theorem 4**.**

Let $k,n,d$ be any positive integers such that $k<d<n.$ Let $\Omega=\{\alpha_{1},\dots,\alpha_{n}\}$, where $\alpha_{i},i=1,\dots,n$ is an element of degree $p_{i}$ over $\mathbb{F}_{p}$ and $p_{i}$ is the $i$th smallest prime that satisfies (7). The code $\mathcal{C}:=\text{\rm RS}_{\mathbb{K}}(n,k,\Omega)$ achieves the cut-set bound for the repair of any single node from any $d$ helper nodes. In other words, $\mathcal{C}$ is an $(n,k)$ MSR code with repair degree $d$.

Remark 2**.**

The code constructions in this paper rely on the condition of the form $\alpha_{i}\not\in\mathbb{F}_{q}(\alpha_{j},j\neq i),i=1,\dots,n$ (in this section we also require that the extension degree $[\mathbb{F}:F_{i}]\equiv 1\text{~{}mod~{}}s,i=1,\dots,n$). The most efficient way to accomplish this in terms of the value of sub-packetization $l$ is to take the extension degrees to be the smallest (distinct) primes, and this is the underlying idea behind the code constructions presented in this paper.

Proof:

Our repair scheme of the $i$-th node is performed over the field $F_{i}$. More specifically, for every $i\in[n]$, we explicitly construct a vector space $S_{i}$ over the field $F_{i}$ such that

[TABLE]

where $S_{i}\alpha:=\{\gamma\alpha:\gamma\in S_{i}\}$, and the operation $+$ is the Minkowski sum of sets, $T_{1}+T_{2}:=\{\gamma_{1}+\gamma_{2}:\gamma_{1}\in T_{1},\gamma_{2}\in T_{2}\}.$ Note that the sum in (9) is in fact a direct sum since the dimension of each summand is $p_{i}$, and $[\mathbb{K}:F_{i}]=sp_{i}$. We will describe a construction of $S_{i}$ and prove that $S_{i}$ satisfies (9) in Lemma 1 later in this section. For now let us assume that we have such vector spaces $S_{i},i=1,2,\dots,n$ and continue the proof of the theorem.

Suppose that we want to repair the $i$-th node from a subset $\mathcal{R}\subseteq[n]\backslash\{i\}$ of $|\mathcal{R}|=d$ helper nodes. Let $h(x)$ be the annihilator polynomial of the set $\{\alpha_{j}:j\in[n]\setminus(\mathcal{R}\cup\{i\})\}$, i.e.,

[TABLE]

By (3) the dual code of $\mathcal{C}$ is $\mathcal{C}^{\bot}=\text{\rm GRS}_{\mathbb{K}}(n,n-k,\Omega,v)$ where the coefficients $v=(v_{1},\dots,v_{n})\in(\mathbb{K}^{*})^{n}$ are nonzero. Clearly, $\deg(x^{t}h(x))\leq s-1+n-(d+1)<n-k$ for all $t=0,1,\dots,s-1,$ so for any such $t$ we have

[TABLE]

These $s$ dual codewords will be used to recover the $i$-th coordinate. Let $c=(c_{1},\dots,c_{n})\in\mathcal{C}$ be a codeword, and let us construct a repair scheme for the coordinate (node) $c_{i}$ using the values $\{c_{j}:j\in\mathcal{R}\}$. Rewrite (11) as follows:

[TABLE]

Let $e_{1},\dots,e_{p_{i}}$ be an arbitrary basis of the subspace $S_{i}$ over the field $F_{i}$. From (12) we obtain the following system of $sp_{i}$ equations:

[TABLE]

Let $\operatorname{tr}_{i}:=\operatorname{tr}_{\mathbb{K}/F_{i}}$ be the trace map to the subfield $F_{i}$. From the last set of equations we have

[TABLE]

Arguing as in (6), let us write (13) in the following form:

[TABLE]

where the second equality follows from (10) and the third follows from the fact that the trace mapping $\operatorname{tr}_{i}$ is $F_{i}$-linear, and that $\alpha_{j}\in F_{i}$ for all $j\neq i$.

As before, to recover $c_{i}$, we download the following $p_{i}$ symbols of $F_{i}$ from each helper node $c_{j},j\in\mathcal{R}$:

[TABLE]

These field elements suffice to recover the node $c_{i}$. Indeed, according to (14), we can calculate the values of $\operatorname{tr}_{i}(e_{m}\alpha_{i}^{t}v_{i}h(\alpha_{i})c_{i})$ for all $t=0,\dots,s-1$ and all $m=1,\dots,p_{i}$ from the set of elements in (15). By definition, $e_{1},\dots,e_{p_{i}}$ is a basis of the subspace $S_{i}$ over the field $F_{i}$. According to (9), $\mathbb{K}=S_{i}+S_{i}\alpha_{i}+\dots+S_{i}\alpha_{i}^{s-1}$. Therefore, the set $\{e_{m}\alpha_{i}^{t}:t=0,\dots,s-1;\,m=1,\dots,p_{i}\}$ forms a basis of $\mathbb{K}$ over $F_{i}$ and so does the set $\{e_{m}\alpha_{i}^{t}v_{i}h(\alpha_{i}):t=0,\dots,s-1;\,m=1,\dots,p_{i}\}$ (recall that $v_{i}\cdot h(\alpha_{i})\neq 0$). Hence the mapping

[TABLE]

is a bijection. This means that $c_{i}$ is uniquely determined by the set of values $\{\operatorname{tr}_{i}(e_{m}\alpha_{i}^{t}v_{i}h(\alpha_{i})c_{i}),m=1,\dots,p_{i};t=0,\dots,s-1\}$, validating our repair scheme.

It is also clear that the construction meets the cut-set bound. Indeed, $c_{j}\in\mathbb{K}$ for all $j$ and $[\mathbb{K}:F_{i}]=sp_{i},$ so the amount of information required from each helper node (15) is exactly $(1/s)$th fraction of its contents.

This completes the proof of Theorem 4. ∎

In the proof above we assumed the existence of the vector space $S_{i}$ that satisfies (9) for all $i\in[n]$. In the next lemma we construct such a space and establish its properties.

For a vector space $V$ over a field $F$ and a set of vectors $A=(a_{1},\dots,a_{l})\subset V$, let $\operatorname{Span}_{F}(A)=\{\sum_{i=1}^{l}\gamma_{i}a_{i},\gamma_{i}\in F\}$ be the span of $A$ over $F$.

Lemma 1**.**

Let $\beta$ be a generating element of $\mathbb{K}$ over $\mathbb{F}=\mathbb{F}_{p}(\alpha_{1},\dots,\alpha_{n}).$ Given $i\in[n]$, define the following vector spaces over $F_{i}$:

[TABLE]

Then

[TABLE]

Proof:

Let $K:=S_{i}+S_{i}\alpha_{i}+\dots+S_{i}\alpha_{i}^{s-1}$. If $K=\mathbb{K},$ then $\dim_{F_{i}}S_{i}=p_{i}$ easily follows. Indeed, by definition $\dim_{F_{i}}S_{i}\leq p_{i}$. On the other hand, $[\mathbb{K}:F_{i}]=sp_{i}$ and $K=\mathbb{K}$ together imply that $\dim_{F_{i}}S_{i}\geq p_{i}$.

Let us prove that $K=\mathbb{K}.$ Clearly $K$ is a vector space over $F_{i}$, and $K\subseteq\mathbb{K}$. Let us show the reverse inclusion, namely that $\mathbb{K}\subseteq K$. To prove this, recall that $\mathbb{K}$ is a vector space of dimension $s$ over $\mathbb{F}$ (see (8)), and the set $1,\beta,\dots,\beta^{s-1}$ forms a basis, i.e., $\mathbb{K}=\oplus_{u=0}^{s-1}\beta^{u}\mathbb{F}$. Thus, the lemma will be proved if we show that $\beta^{u}\mathbb{F}\subseteq K$ for all $u=0,1,\dots,s-1.$ To prove this inclusion we will use induction on $u$.

For the induction base, let $u=0$. In this case, we have $\alpha_{i}^{qs}\in S_{i}^{(1)}$ for all $0\leq q<\frac{p_{i}-1}{s}$. Therefore $\alpha_{i}^{qs+j}\in S_{i}^{(1)}\alpha_{i}^{j}$ for all $0\leq q<\frac{p_{i}-1}{s}$. As a result, $\alpha_{i}^{qs+j}\in K$ for all $0\leq q<\frac{p_{i}-1}{s}$ and all $0\leq j\leq s-1$. In other words,

[TABLE]

Next we show that also $\alpha_{i}^{p_{i}-1}\in K$. For every $t=1,\dots,s-1$ we have $0\leq\lfloor\frac{p_{i}-1-t}{s}\rfloor<\frac{p_{i}-1}{s}$. As a result,

[TABLE]

We obtain, for each $t=1,\dots,s-1,$

[TABLE]

At the same time,

[TABLE]

The last two statements together imply that

[TABLE]

Combining this with (16), we conclude that $\alpha_{i}^{t}\in K$ for all $t=0,1,\dots,p_{i}-1$. Recall that $1,\alpha_{i},\dots,\alpha_{i}^{p_{i}-1}$ is a basis of $\mathbb{F}$ over $F_{i}$, and that $K$ is a vector space over $F_{i}$, so $\mathbb{F}\subseteq K$. This establishes the induction base.

Now let us fix $u\geq 1$ and let us assume that $\beta^{u^{\prime}}\mathbb{F}\subseteq K$ for all $u^{\prime}<u.$ To prove the induction step, we need to show that $\beta^{u}\mathbb{F}\subseteq K$. Mimicking the argument that led to (16), we can easily show that

[TABLE]

Let us show that (17) is also true for $t=p_{i}-1,$ i.e., that $\beta^{u}\alpha_{i}^{u+p_{i}-1}\in K$. For every $1\leq t\leq s-1-u$, we have $0\leq\lfloor\frac{p_{i}-1-t}{s}\rfloor<\frac{p_{i}-1}{s}$. As a result,

[TABLE]

Therefore, for all such $t$

[TABLE]

By the induction hypothesis, $\beta^{u^{\prime}}\mathbb{F}\subseteq K$ for all $u^{\prime}=0,1,\dots,u-1.$ As a result,

[TABLE]

At the same time,

[TABLE]

Combining (18), (19) and (20), we obtain

[TABLE]

Now on account of (17) we can conclude that $\beta^{u}\alpha_{i}^{u+t}\in K$ for all $t=0,1,\dots,p_{i}-1.$ Therefore, $\beta^{u}\mathbb{F}\subseteq K$. This establishes the induction step and completes the proof of the lemma. ∎

The value of sub-packetization of the constructed codes is given in the following obvious proposition.

Proposition 1**.**

The sub-packetization of our construction is $l=[\mathbb{K}:\mathbb{F}_{p}]=s\prod_{i=1}^{n}p_{i}$, where $p_{i}$’s are the smallest $n$ distinct primes satisfying (7).

The proof follows immediately from the fact that the repair of the $i$-th coordinate is performed over the field $F_{i}$, so the repair field of our construction is $\cap_{i=1}^{n}F_{i}=\mathbb{F}_{p}$. To estimate the asymptotics of $l$ for $n\to\infty,$ recall that our discussion of Dirichlet’s prime number theorem in Sec. I-C above implies that, for fixed $s$, $l=e^{(1+o(1))n\log n}$. This proves the upper bound in (2).

IV A lower bound on the sub-packetization of scalar linear MSR codes

In this section we prove a lower bound on the sub-packetization value $l$ of $(n,k)$ scalar linear MSR codes, which implies that $l\geq e^{(1+o(1))k\log k}$. In contrast, for MSR array codes, a much smaller sub-packetization value $l=r^{\lceil n/(r+1)\rceil}$ is achievable [18]. This shows that limiting oneself to scalar linear codes necessarily leads to a much larger sub-packetization, and constructing such codes in real storage systems is even less feasible than their array code counterparts. The main result of this section is the following theorem:

Theorem 5**.**

Let $F=\mathbb{F}_{q}$ and $E=\mathbb{F}_{q^{l}}$ for a prime power $q$. Let $d$ be an integer between $k+1$ and $n-1$. Let ${\mathcal{C}}\subseteq E^{n}$ be an $(n,k)$ scalar linear MDS code with a linear repair scheme over $F.$ Suppose that the repair bandwidth of the scheme achieves the cut-set bound with equality for the repair of any single node from any $d$ helper nodes. Then the sub-packetization $l$ is at least

[TABLE]

where $p_{i}$ is the $i$-th smallest prime.

As discussed above in Sec. I-C, this theorem implies the asymptotic lower bound $l\geq e^{(1+o(1))k\log k}.$

In the proof of Theorem 5, we will need the following auxiliary lemmas.

Lemma 2**.**

(Subfield criterion [21, Theorem 2.6])* Each subfield of the field ${\mathbb{F}}_{p^{n}}$ is of order $p^{m},$ where $m|n.$ For every positive divisor $m$ of $n$ there exists a unique subfield of ${\mathbb{F}}_{p^{n}}$ that contains $p^{m}$ elements.*

Lemma 3**.**

Let $E$ be an extension field of ${\mathbb{F}}_{q}$ and let $\alpha_{1},\dots,\alpha_{n}\in E.$ Then

[TABLE]

where $d_{i}=[{\mathbb{F}}_{q}(\alpha_{i}):{\mathbb{F}}_{q}].$

Proof: Obvious.

Lemma 4**.**

Let $a_{1},a_{2},\dots,a_{n}\in F^{m}$ and $b_{1},b_{2},\dots,b_{n}\in F^{m}$ be two sets of vectors over a field $F$, and let $A$ and $B$ denote their spans over $F$. Let $c_{i}=a_{i}+b_{i},i=1,\dots,n$ then

[TABLE]

The lemma follows immediately from the fact that, for any two subspaces $A$ and $B$ of a linear space,

[TABLE]

In the next lemma $\mathcal{S}_{F}(\cdot)$ refers to the row space of the matrix argument over the field $F$.

Lemma 5**.**

Let $E$ be an extension of a finite field $F$ of degree $l$. Let $A=(a_{i,j})$ be an $m\times n$ matrix over $E$. Then

[TABLE]

Moreover, if (23) holds with equality, then for every $\mathcal{J}\subseteq[n],$

[TABLE]

where $A_{\mathcal{J}}$ is the restriction of $A$ to the columns with indices in $\mathcal{J}$.

Proof:

Inequality (23) is an immediate consequence of Lemma 4. Indeed, suppose that $n=2$ and view the $i$th row of $A$ as the sum of two $2$-dimensional vectors over $E$, namely $(a_{i,1}|0)$ and $(0|a_{i,2}),i=1,\dots,m;$ then (23) is the same as (21). The extension to $n>2$ follows by straightforward induction.

Now let us prove the second part of the claim. Suppose that

[TABLE]

Then for every $\mathcal{J}\subseteq[n],$

[TABLE]

But according to (23),

[TABLE]

Therefore

[TABLE]

This completes the proof of the lemma. ∎

Now we are ready to prove Theorem 5.

Proof of Theorem 5: Let $\mathcal{C}$ be an $(n,k)$ MSR code with repair degree $d.$ By puncturing the code $\mathcal{C}$ to any $d+1$ coordinates, we obtain a $(d+1,k)$ MSR code with repair degree $d$. Therefore without loss of generality below we assume that $d=n-1$.

Let $H=[M|I_{r}]$ be the parity-check matrix of the code ${\mathcal{C}}$ over $E$, written in systematic form, where $M$ is an $r\times k$ matrix and $I_{r}$ is the $r\times r$ identity matrix. Let $h_{ij}$ be the entry of $H$ in position $(i,j).$ Since ${\mathcal{C}}$ is an MDS code, every square submatrix of $M$ is invertible. In particular, every entry of $M$ is nonzero, so without loss of generality we may assume that $h_{1,j}=1,j=1,2,\dots,k.$ Since $d\geq k+1$, we also have $n\geq k+2$, and therefore $H$ contains at least two rows.

The theorem will follow from the following claim.

Claim 1**.**

For $j=1,\dots,k-1$ define $\alpha_{j}:=\frac{h_{2,j}}{h_{2,k}}$. Then for every $j=1,\dots,k-1$,

[TABLE]

In other words, $\alpha_{j}$ is not generated by the remaining $\alpha_{i}$’s over $\mathbb{F}_{q}$.

We first show that this claim indeed implies the theorem. Let $d_{i}=[\mathbb{F}_{q}(\alpha_{i}):\mathbb{F}_{q}]$ be the degree of the field extension generated by $\alpha_{i}$. We prove by contradiction that for all $j=1,2,\dots,k-1$, $d_{j}$ does not divide $\operatorname{lcm}(d_{i}:i\in\{1,2,\dots,k-1\}\setminus\{j\})$. Suppose the contrary, i.e., that there is a $j$ such that $d_{j}|\operatorname{lcm}(d_{i}:i\in\{1,2,\dots,k-1\}\setminus\{j\}).$ According to Lemma 3,

[TABLE]

Then by Lemma 2, there is a subfield

[TABLE]

such that $[F_{j}:\mathbb{F}_{q}]=d_{j}$. Notice that $E=\mathbb{F}_{q^{l}}$ contains all $\alpha_{u},u=1,2,\dots,k-1$. So both $F_{j}$ and $\mathbb{F}_{q}(\alpha_{j})$ are subfields of $E$, and they have the same order $q^{d_{j}}$. Consequently, $\mathbb{F}_{q}(\alpha_{j})=F_{j}$. Then from (26) we conclude that \alpha_{j}\in\mathbb{F}_{q}\big{(}\big{\{}\alpha_{i}:i\in\{1,2,\dots,k-1\}\setminus\{j\}\big{\}}\big{)}, which contradicts (25). Thus, our assumption is wrong, and $d_{j}\not|\,\operatorname{lcm}(d_{i}:i\in\{1,2,\dots,k-1\}\setminus\{j\})$. As an immediate corollary,

[TABLE]

Thus we have shown that this claim indeed implies the theorem. Now let us prove the claim.

Proof of the Claim: Consider the repair of the $j$-th node of the code $\mathcal{C}$ for some $j\in\{1,2,\dots,k-1\}$. Since $\mathcal{C}$ can be viewed as an $(n,k,n-1,l)$ MSR code with a linear repair scheme over $\mathbb{F}_{q}$, node $c_{j}$ can be repaired by downloading $(n-1)l/r$ symbols of $\mathbb{F}_{q}$ from all the remaining nodes $\{c_{i}:i\in[n]\setminus\{j\}\},$ where $r=n-k.$ Therefore by Theorem 1, there exist $l$ codewords

[TABLE]

such that

[TABLE]

Since $H$ is a generator matrix of $\mathcal{C}^{\perp}$, for each $t=1,2,\dots,l$ there is a column vector $b_{t}\in E^{r}$ such that $(c_{t,1},c_{t,2},\dots,c_{t,n})=b_{t}^{T}H$. We define an $l\times r$ matrix $B$ over the field $E$ as $B=[b_{1}b_{2}\dots b_{l}]^{T}$. We claim that the $\mathbb{F}_{q}$-rank of the row space of $B$ is $l$. Indeed, assume the contrary, then there exists a nonzero vector $w\in\mathbb{F}_{q}^{l}$ such that $wB=0$. Therefore,

[TABLE]

This implies that $w(c_{1,j},c_{2,j},\dots,c_{l,j})^{T}=0,$ contradicting (27). Thus we conclude that $B$ has $l$ linearly independent rows over $\mathbb{F}_{q}$.

Now we want to show that there exists an $l\times l$ invertible matrix $A$ over $\mathbb{F}_{q}$ such that the matrix $AB$ is an $r\times r$ block-diagonal matrix $\operatorname{Diag}(a_{1},\dots,a_{r})$, where each block $a_{i}$ is formed of a column vector of length $\frac{l}{r}$. In other words, by performing elementary row operations over $\mathbb{F}_{q}$, $B$ can be transformed into an $r\times r$ block-diagonal matrix $\operatorname{Diag}(a_{1},\dots,a_{r})$. Indeed, for $i\in[n]$, let $h_{i}$ be the $i$-th column of the matrix $H$, and define

[TABLE]

By (28), we have

[TABLE]

Since $H$ generates an $(n,r)$ MDS code, for any subset of indices $\mathcal{J}\subseteq[n]$ of size $|\mathcal{J}|=r$, the matrix $H_{\mathcal{J}}$ is of full rank. Therefore, the $l\times r$ matrix $BH_{\mathcal{J}}$ satisfies the conditions

[TABLE]

where the last inequality follows from Lemma 5. Summing both sides of (30) over all subsets $\mathcal{J}\subseteq[n]\backslash\{j\}$ of size $|\mathcal{J}|=r$, we obtain that

[TABLE]

This implies that the inequality above is in fact an equality, and therefore, on account of (30) for every subset $\mathcal{J}\subseteq[n]\backslash\{j\},|\mathcal{J}|=r$ we have

[TABLE]

From (32) we obtain that for all $i\in[n]\setminus\{j\}$

[TABLE]

Moreover, since (30) holds with equality, we can use the second part of Lemma 5 to claim that, for $\mathcal{J}\subseteq[n]\backslash\{j\}$ of size $|\mathcal{J}|\leq r,$

[TABLE]

Let us take $\mathcal{J}$ to be a subset of $\{k+1,k+2,\dots,n\}.$ Since the last $r$ columns of $H$ form an identity matrix, (34) becomes

[TABLE]

Now we are ready to prove that by performing elementary row operations over $\mathbb{F}_{q}$, $B$ can be transformed into an $r\times r$ block diagonal matrix $\operatorname{Diag}(a_{1},\dots,a_{r})$, where each block $a_{i}$ is a single column vector of length $\frac{l}{r}$. We proceed by induction. More specifically, we prove that for $i=1,2,\dots,r$, we can use elementary row operations over $\mathbb{F}_{q}$ to transform the first $i$ columns of $B$ into the following form:

[TABLE]

where each $\mathbf{0}$ in the last row of the above matrix is a column vector of length $l(1-\frac{i}{r})$.

Let $i=1.$ According to (35), each column of $B$ has dimension $l/r$ over $\mathbb{F}_{q}$. Thus the induction base holds trivially. Now assume that there is an $l\times l$ invertible matrix $A$ over $\mathbb{F}_{q}$ such that

[TABLE]

where each $\mathbf{0}$ in the last row of this matrix is a column vector of length $l(1-\frac{i-1}{r})$. Let us write the $i$-th column of $AB$ as $(v_{1},v_{2},\dots,v_{l})^{T}$. Since each column of $B$ has dimension $l/r$ over $\mathbb{F}_{q}$, $(v_{1},v_{2},\dots,v_{l})^{T}$ also has dimension $l/r$ over $\mathbb{F}_{q}$. Since the last $l(1-\frac{i-1}{r})$ rows of the matrix $AB_{[i-1]}$ are all zero, we can easily deduce that

[TABLE]

By (35), $\dim(\mathcal{S}_{\mathbb{F}_{q}}(AB_{[i]}))=\dim(\mathcal{S}_{\mathbb{F}_{q}}(B_{[i]}))=\frac{il}{r}$. As a result,

[TABLE]

In other words, $(v_{(i-1)l/r+1},v_{(i-1)l/r+2},\dots,v_{l})$ contains a basis of the set $(v_{1},v_{2},\dots,v_{l})$ over $\mathbb{F}_{q}$. This implies that we can use elementary row operations on the matrix $AB$ to eliminate all the nonzero entries $v_{m}$ for $m\leq(i-1)l/r$, and thus obtain the desired block-diagonal structure for the first $i$ columns. This establishes the induction step.

We conclude that there exists an $l\times l$ invertible matrix $A$ over $\mathbb{F}_{q}$ such that $AB=\operatorname{Diag}(a_{1},\dots,a_{r})$, where each block $a_{i}$ is a single column vector of length $\frac{l}{r}$. For $u\in[r]$, let $A_{u}$ be the vector space spanned by the entries of $a_{u}$ over $\mathbb{F}_{q}$. According to (33), for all $i\in[n]\setminus\{j\}$

[TABLE]

Since

[TABLE]

for all $i\in[n]\setminus\{j\}$ we have

[TABLE]

Since each column of $B$ has dimension $l/r$ over $\mathbb{F}_{q}$, $A_{u}$ also has dimension $l/r$ over $\mathbb{F}_{q}$ for every $u\in[r]$. Recall that $h_{u,i}\neq 0$ for all $u\in[r]$ and all $i\in[k]$. Thus

[TABLE]

for all $u=1,\dots,r$ and $i\in[k]\setminus\{j\}.$ Therefore,

[TABLE]

Since $h_{1,i}=1$ for all $i=1,2,\dots,k$, we have

[TABLE]

Equivalently,

[TABLE]

By definition $A_{2}$ is a vector space over $\mathbb{F}_{q}$, so

[TABLE]

On the other hand,

[TABLE]

while

[TABLE]

Equations (38) and (39) together imply that the vector spaces $A_{1}h_{1,j},A_{2}h_{2,j},\dots,A_{r}h_{r,j}$ are pairwise disjoint. In particular, $A_{1}\cap A_{2}h_{2,j}={0}.$ On account of (36), we therefore have $A_{2}h_{2,k}\cap A_{2}h_{2,j}={0}.$ This implies that $A_{2}\alpha_{j}\neq A_{2}$. By (37), we conclude that $\alpha_{j}\notin\mathbb{F}_{q}(\{\alpha_{i}:i\in\{1,2,\dots,k-1\}\setminus\{j\}\})$. This completes the proof of the claim.