Locally Repairable Convolutional Codes with Sliding Window Repair
Umberto Mart\'inez-Pe\~nas, Diego Napp

TL;DR
This paper introduces Locally Repairable Convolutional Codes (LRCCs) that enable efficient local and sliding-window global repair in distributed storage, achieving optimal erasure correction and flexibility through adjustable parameters.
Contribution
The work presents a novel class of LRCCs with adjustable window parameters, a Singleton-type bound for their column distances, and an explicit construction of partial MDP codes based on sum-rank distance convolutional codes.
Findings
LRCCs enable local and global erasure repair with adjustable parameters.
Achieved Singleton-type bound for column distances in LRCCs.
Constructed partial MDP codes that attain the bound for certain parameters.
Abstract
Locally repairable convolutional codes (LRCCs) for distributed storage systems (DSSs) are introduced in this work. They enable local repair, for a single node erasure (or more generally, erasures per local group), and sliding-window global repair, which can correct erasure patterns with up to erasures in every window of consecutive blocks of nodes, where is the th column distance of the code. The parameter can be adjusted, for a fixed LRCC, according to different catastrophic erasure patterns, requiring only to contact nodes, plus less than other nodes, in the storage system, where is the memory of the code. A Singleton-type bound is provided for . If it attains such a bound, an LRCC can correct the same number of catastrophic erasures in a window…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Locally Repairable Convolutional Codes with
Sliding Window Repair
Umberto Martínez-Peñas [email protected]; [email protected] Dept. of Electrical & Computer Engineering, University of Toronto, Canada
Diego Napp [email protected] Dept. of Mathematics, University of Alicante, Spain
Abstract
Locally repairable convolutional codes (LRCCs) for distributed storage systems (DSSs) are introduced in this work. They enable local repair, for a single node erasure (or more generally, erasures per local group), and sliding-window global repair, which can correct erasure patterns with up to erasures in every window of consecutive blocks of nodes, where is the th column distance of the code. The parameter can be adjusted, for a fixed LRCC, according to different catastrophic erasure patterns, requiring only to contact nodes, plus less than other nodes, in the storage system, where is the memory of the code. A Singleton-type bound is provided for . If it attains such a bound, an LRCC can correct the same number of catastrophic erasures in a window of length as an optimal locally repairable block code of the same rate and locality, and with block length . In addition, the LRCC is able to perform the flexible and somehow local sliding-window repair by adjusting . Furthermore, by adjusting and/or sliding the window, the LRCC can potentially correct more erasures in the original window of nodes than an optimal locally repairable block code of the same rate and locality, and length . Finally, the concept of partial maximum distance profile (partial MDP) codes is introduced. Partial MDP codes can correct all information-theoretically correctable erasure patterns for a given locality, local distance and information rate. An explicit construction of partial MDP codes whose column distances attain the provided Singleton-type bound, up to certain parameter , is obtained based on known maximum sum-rank distance convolutional codes.
Keywords: Convolutional Codes, Distributed Storage, Locally Repairable Codes, Locally Repairable Convolutional Codes, Sliding-Window Repair, Sum-Rank Metric.
1 Introduction
Locally repairable codes (LRCs) [11] are an important class of codes for Distributed Storage Systems (DSSs), since they allow to repair a single node by contacting and downloading the content of a small number (called locality) of other nodes (in contrast with MDS codes), while still being able to repair a large number of nodes in case of catastrophic erasures (in contrast with Cartesian products). LRCs are thus natural hybrids between MDS codes and Cartesian products of codes that enjoy both global and local erasure-correction capabilities simultaneously, given by global and local distances, respectively. Note that repair is typically used interchangeably with erasure correction in the storage literature. We will use both terms throughout this work.
LRCs have already been implemented in practice (see [13, 31] for instance). Optimal LRCs (meaning LRCs attaining optimal global distance for a given locality, local distance and information rate) for general parameters and field sizes that are linear in the code length were first obtained in [32]. LRCs capable of correcting all information-theoretically correctable global erasure patterns, for a given locality, local distance and information rate, were introduced in [4, 10] (where they are called partial MDS and maximally recoverable LRCs, respectively). As expected, maximally recoverable LRCs also attain optimal global distance. However, they can correct strictly more global erasure patterns than general optimal LRCs (see Remark 2) for the same parameters. Constructions of maximally recoverable LRCs with relatively small field sizes for general parameters have been given in [8, 24] (see also the references therein).
On the other hand, it is shown in [33] that maximum distance profile (MDP) convolutional codes provide an interesting alternative to MDS block codes since they admit sliding-window erasure correction: They can correct any erasure pattern such that there are no more than erasures in any consecutive blocks of symbols (that is, consecutive symbols), where is the rate of the code (see [33, Th. 3.1] or Fig. 2). Furthermore, the correction is performed somehow locally by sliding recursively the window of blocks, and the parameter may vary arbitrarily up to a certain constant determined by the degree (thus memory) of the convolutional code (see (2)). Therefore MDP convolutional codes already enable certain local and flexible repair, since the window size can be chosen according to how catastrophic the erasure pattern is. Moreover, by adjusting and/or sliding a window of blocks (see Fig. 1), an MDP code can potentially correct in a window of size more erasures than an MDS block code of the same rate and of block length . Unfortunately, in case of one single node erasure (most common case), sliding-window repair with still requires contacting and downloading the content of extra symbols, where is the memory of the code, due to its convolutional nature.
Motivated by the discussion in the previous paragraph, we introduce in this work locally repairable convolutional codes (LRCCs). When being optimal in terms of global distance or maximal recoverability, LRCCs can repair a single node (or more generally, erasures per local group) by contacting (or even ) other nodes and simultaneously enable sliding-window repair (see Fig. 4), which can be set up flexibly according to different catastrophic erasure patterns (see Figs. 4 and 5), and which can potentially correct in a window of size more erasures than an optimal or maximally recoverable locally repairable block code of the same rate and locality, and of block length (see Fig. 1).
LRCCs also enable encoding and storing an unrestricted sequence of files, while locality remains constant and encoding and sliding-window repair complexities are all bounded (by the memory of the code). Furthermore, LRCCs can easily be turned into block codes by converting them into tail-biting convolutional codes, while the properties described above still hold.
We now illustrate with Example 1 and Fig. 1 the main advantages of LRCCs over block LRCs. For fairness, we compare optimal LRCCs with optimal LRCs.
Example 1**.**
Consider a convolutional code that encodes a stream of file vectors over a finite field , each of length , into a stream of encoded vectors, each of length .
Assume a node in the storage system stores a symbol over , and call block each set of coordinates supporting each encoded vector. In this example, each block forms a local group. If the code has locality and local distance (Section 3), it means that a single node erasure ( node erasures) in each block may be repaired by only contacting the other nodes in that block.
The code can correct erasure patterns with up to erasures in every window of consecutive blocks of symbols, where can be adjusted. Assume that an erasure pattern as in Fig. 1 occurs, with erasures in a given window of blocks. If the LRCC has optimal th column distance (as in Corollaries 1 and 2), then , for , and the code cannot correct such erasures considering such a window. Furthermore, an optimal block LRC with block length , dimension , locality and local distance , also has global distance (see [11, Eq. (1)]), hence it cannot correct that erasure pattern either.
However, we may adjust the window for the LRCC. Consider instead windows of length , as in Fig. 1. If the LRCC also has optimal th column distance (as in Corollaries 1 and 2), then for . Observe that now every window of consecutive blocks of symbols contains at most erasures. Therefore, the LRCC may correct such an erasure pattern by sliding the new adjusted window of length , whereas the optimal block LRC as in the previous paragraph cannot.
The disadvantage of the LRCC is that, in order to perform such an erasure correction, we need to read the content (which needs to be correct) of the blocks of symbols previous to such window (see Fig. 1), where is the memory of the LRCC.
Example 2**.**
Consider now a LRCC with locality and local distance (Section 3). Assume also that the code has memory and degree (Subsection 2.1) and that it is an optimal LRCC (as in Corollaries 1 and 2). As in Example 1, such an LRCC can correct the same number of erasures in any window consisting of consecutive blocks ( symbols) as an optimal block LRC of the same rate (), same locality (), same local distance () and total length . In both cases, such a number of erasures is (see Theorem 2 and [11, Eq. (1)], respectively).
However, the LRCC may correct any erasure pattern with up to erasures in any window of consecutive nodes, for all . This only requires reading and downloading the content of the remaining nodes in that window (that is, symbols), plus another previous nodes (see Fig. 5). For instance, for , we may repair any erasure pattern with up to erasures in any consecutive nodes ( blocks of nodes), by contacting nodes in that window, plus another previous nodes. The optimal LRC, in contrast, would always require contacting other nodes in order to repair any erasures in blocks of nodes, since at least one of these blocks contains erasures, which cannot be repaired locally.
Therefore, adjusting the window size when using an LRCC also reduces the number of nodes that need to be contacted. Thus sliding-window erasure correction of LRCCs provides a type of erasure correction in between local and global erasure correction.
Our main contributions are the following. We define LRCCs (Definition 10) and provide a Singleton-type bound on their column distances (Theorem 2), which measure the global sliding window repair capability of the code. We later define partial MDP codes (Definition 15), which can correct all information-theoretically correctable erasure patterns for the given local constrains and, in particular, attain the previous bound for as long as possible. We provide in Construction 1 a method for finding partial MDP codes based on outer MSRD convolutional codes (Theorem 4). By plugging in Construction 1 the MSRD convolutional codes from [21], we obtain an explicit family of partial MDP codes (Corollary 2) for general parameters. Their main disadvantage is their big global field size, although local fields are small. However, this is only an issue in terms of computational complexity, since nodes in DSSs typically store large amounts of data. Furthermore, our construction gives some field size to guarantee the existence of partial MDP codes, but constructions over smaller fields may be possible.
To conclude this introduction, we note that the use of streaming or convolutional codes for storage or as LRCs is not new. Binary tail-biting convolutional codes were proposed as LRCs in [7, 36], but sliding-window repair was not considered. Locality properties of more general (but still binary) convolutional codes were recently considered in [14]. However, LRCCs and sliding-window repair as considered in this work were not treated in [14]. Rateless streaming codes (e.g. Fountain codes [5]) are an interesting alternative to MDS block codes for global repair in DSSs (see [20, Ch. 50]), since they generally achive low redundancy and enable global erasure correction with complexity of XOR operations (products in ) or even less, for encoded symbols. Locally repairable Fountain codes were proposed in [3]. However, their locality is of order (unbounded), for encoded symbols, and they do not enable sliding-window global repair.
The remainder of the paper is organized as follows. In Section 2, we collect some preliminaries on convolutional codes. In Section 3, we introduce LRCCs and give a Singleton-type bound on their column distances, which determine the sliding-window erasure-correction capability of LRCCs. In Section 4, we show how to obtain LRCCs with arbitrary and small-field local codes and optimal global column distances (in view of the previous bound) based on codes with optimal column sum-rank distances [21]. In Section 5, we introduce partial MDP convolutional codes, whose sliding windows can correct analogous erasure patterns as partial MDS block codes [4, 10]. We also provide concrete constructions of partial MDP convolutional codes based on the codes in [21]. Finally, in Section 6, we discuss extending our work to considering LRCCs with unequal localities and local distances, and how to turn our LRCCs to tail-biting convolutional codes.
2 Preliminaries on Convolutional Codes
In this section, we collect general definitions and results on convolutional codes that we will use throughout the paper.
Let be a finite field, and denote by the ring of polynomials with coefficients in . Fix a positive integer . We will typically consider and graphically represent a word in as an unrestricted sequence of vectors of length , , where we use the following terminology. A block is each of the consecutive coordinates in that support each vector , being the th block the block containing the coordinates supporting , for . A symbol is each component of the vectors , thus it is an element of . Finally, a node is the abstraction of the storage device that stores a given symbol. Hence, in this work, each block corresponds to nodes storing symbols over .
2.1 Degree and Memory
Recall that, since is a principal ideal domain, every -submodule of is free.
Definition 1**.**
An convolutional code is a (free) -submodule of rank . A generator matrix of the code is a full-rank matrix such that
[TABLE]
For a vector , we define its degree as the maximum degree of its components, which are polynomials in . We say that a generator matrix of is reduced if the sum of the row degrees of is minimum among generator matrices of , where by row degrees we mean the degrees of the rows in .
It follows from Theorem A-2, Item 3, in [25] that if and are the row degrees of a reduced generator matrix and some other generator matrix , respectively, of , then , for . In particular, the set of degrees of one, thus any, reduced generator matrix is an invariant of the convolutional code . Hence the following definition is consistent.
Definition 2**.**
Given an convolutional code , let be the row degrees of one, thus any, of its reduced generator matrices. We define the degree and memory of , respectively, as
[TABLE]
Note that convolutional codes with zero memory (thus zero degree) coincide with (potentially infinite) Cartesian products of a single block code .
2.2 Non-Catastrophic Codes and Parity-Check Matrices
In most results in this work, although not all, we will require convolutional codes to be non-catastrophic or observable, which we now define in terms of basic generator matrices.
Definition 3**.**
Given an convolutional code , we say that a generator matrix of is basic if it has a polynomial right inverse, that is, if there exists such that . We say that is non-catastrophic if it admits a generator matrix that is reduced and basic.
Observe that any reduced and basic generator matrix of a convolutional code satisfies that is full-rank. For many results in this work, we will only need this weaker property.
Using Theorem A-1, Item 5, in [25], and using the vector space over (the field of fractions of ) generated by a non-catastrophic convolutional code, it is easy to see that it admits a polynomial parity-check matrix. This strong property of non-catastrophic codes is what we will need for sliding-window repair, as described in Subsection 2.4.
Lemma 1**.**
If is a non-catastrophic convolutional code, then there exists a full-rank matrix such that
[TABLE]
We call a (polynomial) parity-check matrix of .
2.3 Free and Column Distances
We now recall the main notions of minimum distance of convolutional codes. Given , we define its Hamming weight as
[TABLE]
The free distance, which we now define, gives the correction capability of a convolutional code when considering whole codewords. In other words, there is no maximum degree for a codeword considered by the free distance.
Definition 4**.**
Given an convolutional code , we define its free distance as
[TABLE]
We next define column distances, which give the sliding-window correction capability of a non-catastrophic convolutional code (see the next subsection). This will be the type of distance that we will be interested in for global repair in our locally repairable codes.
Definition 5**.**
Given an convolutional code , with memory and reduced generator matrix , define the th truncated sliding generator matrix as
[TABLE]
for , where if . Define now the th column block code of as
[TABLE]
where the equality holds if is full-rank. Finally, define the th column distance of as
[TABLE]
where note that if , for .
The column distances satisfy the following Singleton bound, which was proven in [9, Prop. 2.2].
Proposition 1** ([9]).**
For an convolutional code with a generator matrix (possibly not reduced) such that is full-rank, and for , it holds that
[TABLE]
Items 1 and 2 in the following proposition follow from [9, Cor. 2.3] and [30, Th. 2.2], respectively.
Proposition 2** ([9, 30]).**
Given a non-catastrophic convolutional code of degree , the following hold:
If , then . 2. 2.
If , then
[TABLE]
The previous proposition motivates the following definition.
Definition 6**.**
We say that an convolutional code is -MDS if it is non-catastrophic and . We say that is maximum distance profile (MDP) if it is non-catastrophic and , where is as in (2).
2.4 Sliding-Window (Global) Repair
As shown in [33, Th. 3.1] and its proof, a non-catastrophic convolutional code may correct any erasure pattern with up to erasures in any tuple , for . Furthermore, it may do so recursively by sliding a window that only involves the symbols in , where . The formal statement is as follows. See also Fig. 2 for a graphical description.
For convenience, we first define erasures formally.
Definition 7**.**
Let be a symbol not belonging to any finite field, and denote . Given and , we say that is the vector with erasures, where , if components of are the symbol , and and coincide in the other components.
We now state [33, Th. 3.1] and part of its proof.
Theorem 1** ([33]).**
Let be a non-catastrophic convolutional code with memory , and fix . Let and let be such that is the vector with at most erasures, for all . Then, for each , the vector can be recursively and uniquely recovered from the tuple
[TABLE]
by solving a system of non-homogeneous equations, whose coefficients are given by a parity-check matrix of (Lemma 1), the symbols in , and the symbols such that , for , and whose unknowns are , for such that , for . To recover , we “slide” the window (3) one position to the right (see Fig. 2).
In the previous theorem, we implicitly assume that for all .
This type of erasure correction may already be considered as local repair, since may be small. Furthermore, the window size is not necessarily restricted, since may be arbitrary. However, setting , we see that correcting one erasure in a single block requires contacting another nodes and downloading their symbols, corresponding to , in order to set up the necessary system of linear equations. Thus, although sliding-window repair enjoys certain local nature, it admits considerable room for improvement. Adding locality inside each block optimally will be our main objective in the rest of the paper.
3 Locality in Convolutional Codes
In this section, we formulate locality for convolutional codes. For this purpose, we define the following two types of restrictions for a convolutional code. The first type consists in considering one generic block for arbitrary codewords in the convolutional code.
Definition 8**.**
Given an convolutional code with reduced generator matrix , where is the memory of the code, we define its associated block code as
[TABLE]
Note that by the second equality, the definition of does not depend on the generator matrix of . We now give the second type of restriction, which consists in restricting each block of the convolutional code to some subset of coordinates . Here, we use the notation .
Definition 9**.**
Given an convolutional code and given a non-empty subset , we define the restriction of to as the convolutional code
[TABLE]
Here, if , then denotes the projection of onto the coordinates in . Then if , we use the notation .
For a matrix , we denote by the matrix whose rows are the rows of restricted to .
Observe that if is a generator matrix of , then the rows of generate , although they may not be -linearly independent.
We may now extend the definition of -locality for block codes from [16, Def. 1] to convolutional codes.
Definition 10**.**
We say that an convolutional code has -locality if there exist non-empty sets , for , such that , and
, 2. 2.
,
for . Here, we write instead of for simplicity. Thus, denotes the block code associated (Definition 8) to the restriction (Definition 9) of on .
We say then that is an * locally repairable convolutional code*, or LRCC for short. The set is called the th local group, for , and and are called the locality and local distance of , respectively.
In other words, we consider local groups in each block of symbols, corresponding to terms in a codeword . See Fig. 3 for a graphical example of a LRCC with local groups. In contrast to block codes, local repair with only one local group () per block already outperforms sliding-window repair even when , in terms of total contacted nodes, see Fig. 4.
We state now the local erasure-correction capability of LRCCs. Definition 10 is given so that the following result holds. The proof is straightforward.
Proposition 3**.**
Let be an LRCC with local groups , for . Fix and . For all , if is the vector with at most erasures (see Definition 7), then we may uniquely recover the vector from by using the restricted block code , without contacting nodes or reading symbols outside of in the th block of the convolutional code.
As it was the case for locally repairable block codes, the main goal, given the parameters , , and (and now also and ), is to obtain a corresponding LRCC with maximum global distance properties, which would allow for global erasure correction in case of catastrophic failures. In this work, we consider column distances for “global correction”, since we will focus on sliding-window erasure correction as in Theorem 1. See Fig. 4 for a graphical description of local repair combined with sliding-window global repair.
In the next theorem, we provide a Singleton bound on column distances of LRCCs. As the reader can see, we need to make three assumptions for the general bound, the first being that local groups are pair-wise disjoint and of full length, the second being that divides , and the third is that a smallest possible subset of local groups form an information set of the code. This latter condition is satisfied if the [math]th column distance is optimal, as stated in the theorem.
Theorem 2**.**
Let be an LRCC with a reduced generator matrix such that is full-rank. Then it holds that
[TABLE]
Now assume that , for a positive integer , local groups are pair-wise disjoint (i.e., if ) and of full size , and that there exist of them forming an information set for the -dimensional linear block code , generated by . This latter condition holds if equality is achieved in (4), by Lemma 7 in Appendix A. Then it holds that
[TABLE]
for all .
Proof.
We start by observing that and the block code
[TABLE]
is a -dimensional linear block LRC of length with -localities. Hence the bound (4) is the classical upper bound on the minimum Hamming distance of linear block LRCs [16, Th. 2.1].
We will now prove the bound (5), for , given the assumptions in the theorem. Assume that local groups are pair-wise disjoint, and that , for . Finally, assume, without loss of generality, that the first local groups form an information set for the linear block code .
Let denote the first coordinates in , for . By the -locality of , the set
[TABLE]
is an information set of of size . Hence we may perform row operations on the generator matrix of to obtain a systematic generator matrix of the form
[TABLE]
for matrices and , and where are such that
[TABLE]
is the identity matrix.
Fix now . Using the systematic generator matrix of from (7), we may perform row operations on the th truncated sliding generator matrix from Definition 5 to obtain a row equivalent matrix (i.e., a matrix with the same row space) of the form
[TABLE]
such that
[TABLE]
for matrices and , for , and where denotes the zero matrix.
Now, let be the first row of the matrix from (8). By (9), we have that
[TABLE]
for vectors , , and where denotes the zero vector of length .
Clearly , since it is a linear combination of rows of the matrix from Definition 5, and its first block of components is nonzero, that is, .
Finally, since is an LRCC, by Item 2 in Definition 10, we deduce that
[TABLE]
where is the zero vector of length .
Therefore, we conclude that
[TABLE]
and we are done. ∎
In the next section, we show how to explicitly construct a non-catastrophic LRCC attaining the previous bound, for all , where is as in (2), for fields of any characteristic but sufficiently large.
Remark 1**.**
Recall that, by Proposition 2, a convolutional code that is -MDS is also -MDS, for all . However, it is not clear whether a code attaining the bound (5) for some implies attaining the bound for . We leave this as an open problem. In any case, Construction 1 below based on a -MSRD convolutional code attains the bound (5) for all .
4 LRCCs based on Sum-Rank Convolutional Codes
In this section, we will show how to construct non-catastrophic LRCCs attaining the bound in Theorem 2, for , using a -MSRD convolutional code (see Definition 13 below). To that end, we will use the notion of sum-rank weight on each block of a convolutional code. Sum-rank weights were first defined in [28] for error correction in multishot network coding (see also [21, 22, 23, 26, 34] and the references therein). They were implicitly considered earlier in the space-time coding literature (see [19, Sec. III]), and they have been first used for locally repairable block codes in [24].
Throughout this section, we will fix a prime power and a positive integer , and we will assume that . Fix an ordered basis of over . For any positive integer , we denote by the corresponding matrix representation map, given by
[TABLE]
where and , for and .
Throughout this section, we will also fix a number of local groups , a locality , and the sum-rank length decomposition . The following definition is given in [28].
Definition 11** ([28]).**
Let , where , for . We define the sum-rank weight of as
[TABLE]
We extend sum-rank weights to convolutional codes as follows.
Definition 12**.**
Given , we define its sum-rank weight as
[TABLE]
Given an convolutional code , we define its sum-rank free distance as
[TABLE]
Finally, we define the th sum-rank column distance of as
[TABLE]
where is as in Definition 5, in particular if , for .
Observe that, for any , where , for , it holds that
[TABLE]
since the rank of a matrix is at most the number of its non-zero columns. Hence, the following result follows immediately from its Hamming-metric counterpart (Propositions 1 and 2).
Proposition 4**.**
Given a non-catastrophic convolutional code , it holds that
[TABLE]
for all . Furthermore, the following hold.
If , then , for . 2. 2.
If , then , where is as in (2).
The previous proposition motivates the following definition.
Definition 13**.**
We say that an convolutional code is -maximum-sum-rank-distance, or -MSRD for short, if it is non-catastrophic and .
We now describe how to construct LRCCs from sum-rank convolutional codes. This construction is inspired by [29, Const. I].
Construction 1**.**
Assume that , and choose:
Outer code: An convolutional code . 2. 2.
Local codes: An MDS block code with generator matrix . 3. 3.
Global code: We define the global code , with , as the convolutional code given by
[TABLE]
where is defined as a block-diagonal matrix with repeated times (recall that and ):
[TABLE]
Observe that if is a generator matrix of , then a generator matrix of is simply given by
[TABLE]
In addition, note that multiplying a vector on the right by a rank- constant matrix preserves the degree of . Hence if is reduced, then so is . It also follows easily that if is basic, then so is . Thus we deduce the following.
Lemma 2**.**
In Construction 1, if is non-catastrophic, then so are and , for any such that and , for . Here, we denote , for .
As it was the case for locally repairable block codes (see [24, Lemma 1]), any LRCC whose local codes are all encoded by the same linear MDS code over the subfield , are necessarily of the form of Construction 1. For this reason, Construction 1 not only is natural, but it is somehow unavoidable.
We may now prove the main result of this section, which states that in Construction 1 has maximum th sum-rank column distance among all non-catastrophic LRCC, for , if is -MSRD.
Theorem 3**.**
In Construction 1, is an LRCC. Furthermore, if and is -MSRD (thus non-catastrophic), then is non-catastrophic and
[TABLE]
for all .
Proof.
First, it follows easily from the definitions and Construction 1 that is an LRCC. The non-catastrophic property is part of Lemma 2. Therefore, we only need to show that (12) holds for , since if is -MSRD, then is -MSRD, for all , by Proposition 4.
Now we need to show that, for any , the non-zero coordinates of are not all inside some pattern of
[TABLE]
erasures in the block of coordinates.
Assume the opposite holds, that is, there exists with all of its non-zero coordinates in an erasure pattern of size . Observe that is a block codeword, and by construction, there exists such that .
Let be the erasure pattern in the th local group in the th block of coordinates, and define , for and . The truncated global codeword after removing all the symbols in such an erasure pattern is by assumption the zero vector, that is,
[TABLE]
where .
Assume for simplicity that , for some integer . Note that we may decompose
[TABLE]
As discussed in the proof of [29, Th. 24], the best-case erasure pattern is obtained when erasures concentrate in the smallest number of local groups. Here, by best-case erasure pattern we mean an erasure pattern whose complement set of coordinates contain the most locally redundant symbols, which means that is the minimum possible. Thus by (14), in the best case we have without loss of generality that , for , , and , for . Since is an MDS code, we have that , for . Therefore, in the best case, we have that
[TABLE]
Define now as the set formed by some coordinates in , for . Define also as any coordinates in , for . Since is an MDS code, we have that , that is, is invertible, for . Therefore, we conclude that
[TABLE]
where the last inequality follows from (13). This is absurd since and
[TABLE]
We conclude that there is no whose non-zero coordinates are all inside some pattern of erasures, hence , and we are done. ∎
We conclude by plugging in Construction 1 the MSRD convolutional codes from [21] (see Appendix B) as outer codes, and applying the previous theorem.
Corollary 1**.**
If , , , , and , then there exists a non-catastrophic LRCC , of degree , satisfying (5) with equality, for , given as in Construction 1, and where is the non-catastrophic -MSRD convolutional code in Appendix B.
Corollary 1 not only shows that the upper bound given in (5) is sharp, but also provides an explicit class of codes that achieves such a bound. Moreover, these codes exist for any characteristic (in particular, when ), and the local code may be arbitrary and with local fields of size , which are small. We may also choose if and local repair would simply consist in XORing. Their main disadvantage is the huge exponent , which is in turn exponential in the degree and in . However, the bound on in the corollary is only a bound, and there are cases when can be chosen much smaller (see Table I in [21]).
5 Partial -MDS and Partial MDP Convolutional Codes
In this section, we introduce partial MDP convolutional codes, analogous to the concept of partial MDS codes, or LRC with maximal recoverability (MR), introduced in [4, 10]. We will conclude by showing that the codes in Corollary 1 are partial MDP.
Definition 14**.**
With notation as in Definition 10, and for , we say that an LRCC is partial -MDS if the following holds: For all such that , for , the restricted convolutional code is non-catastrophic and -MDS (Definition 6), where and .
Some explanations about Definition 14 are in order.
First, we observe that the restricted convolutional code in the previous definition has rank by Lemma 3 below, thus the definition is consistent.
Lemma 3**.**
Let be an LRCC with local groups , for , as in Definition 10. Let be such that , for , and define and . Then the restricted code has rank , or in other words, it is an convolutional code.
In addition, if is a reduced generator matrix of such that is full-rank, then is also full-rank.
Proof.
Let be a generator matrix of . It suffices to prove that the rows of are -linearly independent.
Assume that there exists such that . If , then we have that . Write and fix . We then deduce that , and therefore , for . Since and , we deduce that , for . Now, because , we conclude that .
Thus we have proven that . Since has full rank, we conclude that , and we are done. The statement regarding and is proven following the same lines. ∎
Similar to the case of block codes (replacing -MDS by MDS), the term partial -MDS is motivated by the fact that the column distances attain the bound (5) (see Proposition 5 below), thus they have smaller column distances than those of -MDS codes (this is the price to pay for locality). However, partial -MDS codes as in Definition 14 can be seen as -MDS codes that can be added locality in some optimal sense: We can recover some other -MDS code after removing any (maximal) collection of local parities, not only the added ones. Due to this reason, we gain a considerable flexibility in the erasure patterns that can be corrected (see Fig. 5).
In the block case, partial MDS codes can be equivalently defined as follows: A locally repairable block code is partial MDS if it can correct all erasure patterns that are information-theoretically correctable for the given local constraints and and the given dimension and length . Obviously, if there are no local constraints ( for instance), then being able to correct all information-theoretically correctable erasure patterns is equivalent to being MDS.
See Fig. 5 for a graphical description of sliding-window repair combined with local repair in a partial -MDS convolutional code.
We now show that partial -MDS codes attain the bound (5), hence being optimal LRCCs in terms of column distances. We need a preliminary lemma, which is of interest by itself and which follows directly from Definition 14 and Proposition 2.
Lemma 4**.**
If an LRCC is partial -MDS, then it is partial -MDS, for all .
Proposition 5**.**
If an LRCC is partial -MDS for some , then its column distances attain the bound (5), for all .
Proof.
By the previous lemma, we only need to prove the result for . For such a case, the proof follows exactly the same lines as the proof of Theorem 3, and is left to the reader. ∎
Remark 2**.**
In the block case, the converse is not true. For instance, Tamo-Barg codes [32] are locally repairable codes with optimal global distance, but cannot always be maximally recoverable (partial MDS) by the field-size bound in [12, Eq. (2)]. We conjecture, but do not prove or disprove, that not every LRCC attaining the bound (5), for some , is a partial -MDS convolutional code.
Our next goal is to define partial MDP convolutional codes, which are partial -MDS for the maximum value of . We first need the following lemma, which follows directly from the definitions and Proposition 2.
Lemma 5**.**
Let be an convolutional code. For any , it holds that
[TABLE]
In particular, if is -MDS, then , for and .
We may now define partial MDP convolutional codes.
Definition 15**.**
We say that an LRCC is partial MDP if it is partial -MDS for , where and .
The main purpose of this section is to show that the global code in Construction 1 based on an MSRD outer code (for instance, that in Appendix B) is partial MDP. In particular, we will show the existence of partial MDP codes for general parameters, over any characteristic, for sufficiently large fields.
We first need the following lemma, which is [24, Th. 1]. Observe that we will make use of this lemma in the non-linear case.
Lemma 6** ([24]).**
Recall that . Given a (linear or non-linear) block code , it holds that
[TABLE]
We may now prove the main result of this section.
Theorem 4**.**
In Construction 1, the following hold:
If and is non-catastrophic and -MSRD, then is partial -MDS. 2. 2.
* and .* 3. 3.
If is non-catastrophic and -MSRD, where and , then is partial MDP.
Proof.
We start by proving Item 1. Let be such that (i.e. since in Construction 1), for . If and , then the restricted code is the convolutional code given by
[TABLE]
Since is an MDS block code and , we deduce that is invertible, for . Thus is non-catastrophic by Lemma 2, and moreover by (15) and Lemma 6, we have that
[TABLE]
Hence is -MDS and Item 1 follows.
Now, Item 2 follows from the fact that , and multiplying by the full-rank constant matrix on the right preserves degrees. Finally, Item 3 follows by combining Items 1 and 2. ∎
Finally, by plugging in the previous theorem the -MSRD codes from [21] (see Appendix B) as outer codes in Construction 1, we show the existence of partial MDP convolutional codes.
Corollary 2**.**
If , , , , and , then the convolutional code from Corollary 1 is an partial MDP convolutional code.
Observe that we could have given the previous corollary first, and then deduce Corollary 1 from Proposition 5. However, we have chosen to present our results in this order for simplicity.
6 Further Considerations
6.1 Unequal Localities and Local Distances
Locally repairable codes with unequal localities were introduced independently in [15, 35]. Adding also unequal local distances was first considered in [6]. Essentially, locally repairable codes with unequal localities are those such that the locality and local distance depend on the local group (see Definition 10). In other words, the th local group has locality and local distance , for . We may then modify Definition 10 to include unequal localities and local distances by adding indices to Items 1 and 2:
, 2. 2.
,
for . The main motivation for this type of locally repairable codes is that some nodes may require faster repair or access (hot data), while considering the different localities in general improves the global correction capability of the code.
Finding analogous upper bounds to (5) is a challenging task in general. Such bounds are known when and (see [6, Th. 2] and [17, Th. 2]).
On the other hand, adapting the notion of partial MDS codes to unequal localities is straightforward (see [24, Def. 5]). In addition, it was proven in [24, Th. 2] that MSRD block codes used as outer codes always give partial MDS codes, for any choice of unequal localities and local distances.
All the results in this work hold also for unequal localities and local distances. As in the block case, bounds on the column distances are not straightforward in general. However, Construction 1 with the MSRD codes from Appendix B as outer codes provide partial MDP codes for an arbitrary choice of unequal localities and local distances, just as in the block case. We leave the details to the reader.
6.2 Tail-Biting LRCCs
LRCCs may encode an unrestricted number of information symbols (i.e. files or file components), while locality and sliding-window erasure-correction capability and complexity remain constant. However, truncating an LRCC at a given block implies that, for , the final windows cannot be the initial part of a sliding window consisting of blocks, which could potentially correct erasures. Therefore, in such a truncated LRCC, certain blocks receive a weaker protection against erasures.
To provide equal protection to all blocks, one solution is to terminate the LRCC as a block code by converting it into a tail-biting convolutional code. This simply requires updating the first blocks using the last , in the way they would be encoded if we had used the generator matrix
[TABLE]
where is a reduced generator matrix of the LRCC. In this way, sliding-window repair behaves equally in any window of the same size. However, we always need to have at least consecutive blocks with no erasures in order to get the repair started, although this consecutive blocks may be arbitrary and not necessarily the first . In other words, any consecutive blocks may be considered initial in a tail-biting convolutional code.
Acknowledgement
The first author is supported by The Independent Research Fund Denmark (Grant No. DFF-7027-00053B). The second author is partially supported by the Generalitat Valenciana (Grant No. AICO/2017/128) and the Universitat d’Alacant (Grant No. VIGROB-287).
Appendix A A Lemma on Information Sets of Optimal Block LRCs
In this appendix, we prove the following result on the information sets of optimal block LRCs. Essentially, we follow a simplified version of the proof of [29, Th. 21], using linear LRCs, thus dimensions instead of entropies, and for pair-wise disjoint local groups of size exactly , for -localities.
Lemma 7**.**
Let be a -dimensional block linear LRC with -localities, where we are considering pair-wise disjoint local groups of size exactly : , where if , and , for . Define , and assume that has maximum possible minimum Hamming distance, i.e.,
[TABLE]
Then there are local groups, which we may assume without loss of generality that they are the first of them, , such that
[TABLE]
where denotes the restriction of a block code onto the coordinates in .
Proof.
We proceed as in the proof of [29, Th. 21], and define the following algorithm, which finds a size- subset of local groups satisfying the properties in the lemma. As explained above, this algorithm is the same as that in the proof of [29, Th. 21], but considering only linear LRCs, replacing entropies by dimensions, and considering pair-wise disjoint local groups of size exactly , for -localities.
1: Set and .
2: while do
3: Pick an index .
4: if then
5: Set .
6: Set .
7: else if and s.t. then
8: Set .
9: Set .
10: else
11: end while
12: end if
13: end while
14: return
Now we run the algorithm above. As in the proof of [29, Th. 21], there may be only the following two cases.
Case 1: Assume that the algorithm terminates with the final sets and assigned at lines 5 and 6, respectively. Since the algorithm has terminated at this point, if we consider any , then
[TABLE]
Hence we reassign , and then it must hold that
[TABLE]
since adding a local group may not increase the dimension of the restricted code by more than , since out of coordinates in a local group are redundant.
Case 2: Assume that the algorithm terminates with the final sets and assigned at lines 8 and 9, respectively. In this case, we already have that
[TABLE]
thus, without reassigning , we also deduce that
[TABLE]
In any of the two cases, Case 1 or Case 2, assume that . Following the same steps as in the proof of [29, Th. 21], we have that ()
[TABLE]
This contradicts the optimality of the LRC , hence the case may not happen. Therefore, it must hold that , in both Case 1 and Case 2. Also in both cases, the local groups , for , satisfy the properties of the lemma, i.e.,
[TABLE]
and thus we are done. ∎
Appendix B Known Construction of MSRD Convolutional Codes
In this appendix, we revisit the construction of non-catastrophic MSRD convolutional codes from [21], which is based on the superregular matrices introduced in [1]. To the best of our knowledge, this is the only known construction of MSRD convolutional codes. In addition, they admit general parameters, except that they usually require impractically large field sizes. Acceptable field sizes can be achieved for certain parameters. See Table I in [21] for a few instances.
Fix . As in [1], see also [2], we will restrict ourselves to convolutional codes whose degree satisfies that , for general parameters see [27]. Define and , as in (2). Let be any prime power and assume that
[TABLE]
The field will be then . Let be a primitive normal element over , that is, a primitive element of such that form a basis of over . Such element exists for any finite field extension (see [18]). Define the matrix
[TABLE]
for , where , for . Finally, define the non-catastrophic convolutional code as that with polynomial parity-check matrix
[TABLE]
[TABLE]
where , , and can be given from by the rule
[TABLE]
The following theorem combines [9, Th. 3.1] with [21, Th. 5].
Theorem 5**.**
The convolutional code described above is non-catastrophic, has degree and is -MSRD for any sum-rank length decomposition of .
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] P. Almeida, D. Napp, and R. Pinto. A new class of superregular matrices and MDP convolutional codes. Linear Algebra and its Applications , 439(7):2145–2157, 2013.
- 2[2] P. Almeida, D. Napp, and R. Pinto. Superregular matrices and applications to convolutional codes. Linear Algebra and its Applications , 499:1–25, 2016.
- 3[3] M. Asteris and A. G. Dimakis. Repairable fountain codes. IEEE J. Select. Areas Comm. , 32(5):1037–1047, May 2014.
- 4[4] M. Blaum, J. L. Hafner, and S. Hetzler. Partial-MDS codes and their application to RAID type of architectures. IEEE Trans. Info. Theory , 59(7):4510–4519, July 2013.
- 5[5] J. W. Byers, M. Luby, M. Mitzenmacher, and A. Rege. A Digital Fountain approach to reliable distribution of bulk data. SIGCOMM Comput. Commun. Rev. , 28(4):56–67, October 1998.
- 6[6] B. Chen, S. T. Xia, and J. Hao. Locally repairable codes with multiple ( r i , δ i ) subscript 𝑟 𝑖 subscript 𝛿 𝑖 (r_{i},\delta_{i}) -localities. In Proc. IEEE Int. Symp. Info. Theory , pages 2038–2042, June 2017.
- 7[7] A. Datta. Locally repairable rapid RAID systematic codes — one simple convoluted way to get it all. In Proc. IEEE Info. Theory Workshop , pages 60–64, Nov 2014.
- 8[8] R. Gabrys, E. Yaakobi, M. Blaum, and P. H. Siegel. Constructions of partial MDS codes over small fields. IEEE Trans. Info. Theory , 65(6):3692–3701, Dec 2018.
