Clustered Distributed Data Storage Repairing Multiple Failures

TL;DR

This paper studies how to efficiently repair multiple node failures in clustered storage systems by leveraging cheaper intra-cluster communication.

Contribution

The paper derives a storage-repair bandwidth trade-off and provides explicit code constructions for clustered DSSs.

Findings

Failed nodes can repair more efficiently by downloading data from the same cluster.

Collaboration among failed nodes in the same cluster reduces repair bandwidth.

Explicit codes achieve the minimum storage and minimum bandwidth repair points.

Abstract

A clustered distributed storage system (DSS), also called a rack-aware storage system, is a distributed storage system in which the nodes are grouped into several clusters. The communication between two clusters may be restricted by their connectivity; that is to say, the communication cost between nodes differs depending on their location. As such, when repairing a failed node, downloading data from nodes that are in the same cluster is much cheaper and more efficient than downloading data from nodes in another cluster. In this article, we consider a scenario in which the failed nodes only download data from nodes in the same cluster, which is an extreme and important case that leverages the fact that the intra-cluster bandwidth is much cheaper than the cross-cluster repair bandwidth. Also, we study the problem of repairing multiple failures in this article, which allows for collaboration within the same cluster, i.e., failed nodes in the same cluster can exchange data with each other. We derive the trade-off between the storage and repair bandwidth for the clustered DSSs and provide explicit code constructions achieving two extreme points in the trade-off, namely the minimum storage clustered collaborative repair (MSCCR) point and the minimum bandwidth clustered collaborative repair (MBCCR) point, respectively.

Figures

Full text

1. Introduction

Cloud distributed storage systems (DSSs), built on a huge number of storage nodes, have been applied widely (e.g., Google File System [1], Facebook [2], and Total Recall [3]). In a DSS, failures occur frequently, degrading the system’s reliability. To ensure the reliability of DSSs, erasure coding has been widely used to improve their tolerance of failures.

A class of erasure codes, namely regenerating codes, for cloud distributed storage systems was proposed by Dimakis et al. [4], aiming to reduce the repair bandwidth (or repair traffic) during node failures. More specifically, suppose that a data file is encoded and stored on n nodes, each of which stores $[eqn]$ symbols. A data collector can retrieve the data file by connecting to any k nodes, which is called the * $[eqn]$ recovery* property. When a node failure occurs, the system will trigger a repair process to recover the data on the failed node by downloading data from some other nodes. The total amount of data downloaded from these other nodes is called the repair bandwidth, which consists of $[eqn]$ symbols. The fundamental trade-off between the amount of data stored on each node $[eqn]$ and the repair bandwidth $[eqn]$ was studied in [4]. There are two extreme points on the trade-off curve, namely the minimum storage repair (MSR) point and the minimum bandwidth repair (MBR) point, which minimize the data stored on each node and the repair bandwidth, respectively.

The pioneering work [4] proposed studying a single failure in a cloud distributed storage system. Later, collaborative regenerating codes (CRCs) were proposed [5,6,7] for studying cases when multiple failures happen and are repaired simultaneously in a DSS, which has been shown to improve the repair bandwidth compared to repairing multiple failures sequentially.

The former studies considered the communication cost between the nodes to be homogeneous. However, in a realistic system, the storage nodes may be geographically separated in their locations, and hence the communication cost of downloading data from a geographically close helper node may be quite different from that of downloading from a helper node that is further away geographically. The model of a clustered (or rack-aware) distributed storage system assumes that the nodes are grouped into several clusters (racks) and assumes that the communication cost of downloading data from a node within the same cluster (rack) (which is also called the host cluster below) is much cheaper than downloading data from another cluster (rack). In fact, as shown in [8,9,10], the clustered structure incurs a substantial cross-cluster (cross-rack) bandwidth. The cross-cluster (cross-rack) capacity available per node in the worst case is only 1/5 to 1/20 of the intra-cluster (intra-rack) capacity, which indicates that the cross-cluster (cross-rack) bandwidth is more expensive than the intra-cluster (intra-rack) bandwidth.

Many attempts [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25] have been made to reduce the cross-cluster bandwidth, in which repairing a single failure has been considered. Following these works, cases of clustered structures with multiple failures in each cluster were considered in [26,27,28,29,30,31,32], where the failed nodes downloaded data from other clusters so that the cross-cluster bandwidth applied.

As said, the cross-cluster (cross-rack) bandwidth is more expensive than the intra-cluster (intra-rack) bandwidth, so considering repairing multiple failures within the host cluster without cross-cluster traffic is an economical choice. In this article, we address the problem of repairing multiple failures in clustered storage systems, where multiple failures happen in each cluster. A data collector can retrieve the data file by connecting to any k nodes (from multiple clusters). We suppose the system has no cross-cluster bandwidth, and the target is to minimize the repair bandwidth within the host cluster. The main contributions of this article can be summarized as follows:

• (1)We first study the trade-off between the storage and repair bandwidth for clustered distributed storage systems for multiple failures with zero cross-cluster bandwidth using a conventional maximum-flow, min-cut analysis over an information flow graph;
• (2)We calculate the parameter values for two extreme points on the trade-off curve, namely the minimum storage clustered collaborative repair (MSCCR) point and the minimum bandwidth clustered collaborative repair (MBCCR) point;
• (3)We analyze the repair bandwidth performance using different system parameters at the MSCCR and MBCCR points;
• (4)We also provide explicit constructions to optimize the two extreme points on the trade-off curve, which implies that our constructions are optimal in terms of the repair bandwidth.

This article is organized as follows. Section 2 discusses related works. In Section 3, we briefly describe the clustered collaborative repair problem and introduce the standard notion of an information flow graph for the clustered collaborative repair problem. In Section 4, we give the min-cut bound (an upper bound on the file size) by inspecting the corresponding information flow graph. In Section 5, the two extreme points on the storage bandwidth trade-off curve, i.e., the MSCCR point and the MBCCR point, are identified. In Section 6, we analyze the normalized repair bandwidth at the MSCCR point and the MBCCR point and provide some results of simulations. In Section 7, we give the explicit code constructions for the MSCCR point and the MBCCR point, respectively. Finally, we conclude this article in Section 8 and point out further directions for future research.

2. Related Works

2.1. Collaborative Regenerating Codes

To reduce the repair bandwidth (repair traffic) when multiple failures happen in DSSs, collaborative regenerating codes (CRCs), or precisely fully collaborative regenerating codes, were proposed in [5,6,7]. Multiple failures were repaired simultaneously. The repair bandwidth of collaborative regenerating codes includes the amount of data exchanged between the failed nodes and the amount of data downloaded from the helper nodes. It has been shown [5,6,7] that collaborative repair improves the repair bandwidth compared to that when repairing multiple failures sequentially. Many attempts [33,34,35,36,37,38,39] have been made to construct collaborative regenerating codes at the minimum storage regenerating point and the minimum bandwidth regenerating point.

A related variant of fully collaborative regenerating codes is called partially collaborative regenerating codes [40], which can be viewed as a generalization of the collaborative repair mechanism and the sequential repair mechanism by allowing a subset of failed nodes to join the repair process collaboratively. A few constructions at the MSR point and the MBR point for partially collaborative regenerating codes were proposed in [41,42,43]. The problem of repairing multiple node failures has also been studied under the scenario of centralized repair [44,45], where a centralized node downloads data from some nodes that are still alive, computes the data, and then dispatches the data for all failed nodes.

2.2. Clustered Storage Codes

In a realistic system, the communication costs between storage nodes are heterogeneous when the nodes are geographically separated. The model of a clustered distributed storage system assumes that the nodes are grouped into several clusters. The model assumes that the communication cost of downloading data from a node within the same cluster is much cheaper than downloading data from another cluster.

A case with two clusters was studied in [11,12], and an upper bound on the file size that could be stored was derived. An explicit construction of maximum distance separable (MDS) array codes of rate 1/2 for a two-cluster case was proposed in [14]. A general case with more than two clusters was later studied in [13,14], where a single failed node was repaired by downloading data from the nodes in the host cluster and also from nodes in other clusters directly. An upper bound on the file size was proposed [13], and the existence of optimal repair codes was established via an information flow graph.

Recently, many attempts [15,16,17,18,19,20,21,22,23,24,25] have been made to reduce the cross-cluster bandwidth. In these works, a centralized node (also called a relay node/a compute unit) exists in each cluster, which collects data from the nodes within the cluster and then computes and dispatches data to repair the failed node. The failed node being repaired will download data from the centralized nodes in other clusters, indicating that the helpers are clusters rather than nodes. Hence, the model in these works can be seen as a centralized model, which is different from the model in [11,12,13,14] (which can be seen as a decentralized model), where the failed node being repaired directly downloads the data from nodes in other clusters without processing by the centralized node.

In [15], Hu et al. considered a clustered system with centralized nodes, and they derived an upper bound for the file size and showed the existence of minimum storage codes with optimal repair for the model. In [18], Hou et al. studied codes not only for the minimum storage but also for the minimum bandwidth. In [19], they derived the lower bound on the amount of information that needed to be accessed to repair a single failed node using an arbitrary number of helper racks. In [20], they proposed codes by product-matrix method for the minimum storage. Chen and Barg [21] further proposed a family of minimum storage rack-aware regenerating codes for all admissible parameters but with a high level of sub-packetization. Hou et al. [22] then proposed a family of minimum storage rack-aware regenerating codes that improved the sub-packetization level.

Clustered structures with multiple failures in each cluster were considered in [26,27,28,29,30,31,32]. In particular, Abdrashitov et al. [26] studied the problem of multiple node failures occurring in one cluster, and in their model, the data reconstruction was realized by connecting to any k clusters rather than any k nodes. Gupta and Lalitha [27,28,29] studied the case of multiple node failures occurring in more than one cluster, allowing for collaboration among clusters, where a centralized node existed for each cluster, the same setting as that seen in [15,16,17,18,19,21,22,23,24,25] as mentioned, and then the collaboration happened among the clusters instead of the nodes. Multiple node failures in a single cluster were considered in [30,31,32].

To conclude, we consider the problem of repairing multiple node failures in a clustered distributed storage system without a centralized node. The node repair happens within the host cluster (for the case of zero cross-cluster bandwidth), which is the most desirable setting, as mentioned, i.e., the intra-cluster bandwidth is much cheaper than the cross-cluster bandwidth. Within the cluster, the bandwidth will be further improved by utilizing the properties of collaborative regenerating codes.

3. The System Model

In this section, we will describe the clustered collaborative repair problem formally and the associated information flow graph that represents the problem.

A data object of size M is stored across n nodes, which is equally distributed across L clusters with l nodes each. Every node has a storage capacity of $[eqn]$ . In this clustered distributed storage system, any choice of k nodes allows for object retrieval, where $[eqn]$ (a case where $[eqn]$ will be equivalent to the case in [5,6]). The system triggers a repair process when t failures occur in one cluster, and the t failures are repaired collaboratively in each cluster. Each node being repaired will download a $[eqn]$ amount of data from the $[eqn]$ surviving nodes in its host cluster and exchange a $[eqn]$ amount of data with the other $[eqn]$ nodes being repaired. The repair bandwidth for each node is $[eqn]$ . For ease of reading, some of the frequently used notations are summarized in Table 1.

Following the information flow graph approach from the seminal work [4], we consider the corresponding information flow graph associated with this clustered collaborative repair problem, where the data flow from a source S to a data collector $[eqn]$ . Suppose that a data collector $[eqn]$ connects to a subset of k nodes that are involved in different clusters. Let $[eqn]$ denote the number of nodes connected by $[eqn]$ in cluster r, where $[eqn]$ (for any positive integer a, denote $[eqn]$ ). As such, we have $[eqn]$ . In each cluster r, the connected $[eqn]$ nodes are involved in different phases of repair (each repair phase is a repair process of t nodes), where each phase i involves a group of $[eqn]$ nodes such that $[eqn]$ . Let $[eqn]$ , where $[eqn]$ .

We give an example of the information flow graph for two clusters here. As shown in Figure 1, the file is stored in $[eqn]$ nodes, which are divided into $[eqn]$ clusters, and each cluster has $[eqn]$ nodes. In each cluster, $[eqn]$ nodes are being repaired simultaneously. The data collector can retrieve the file by connecting to $[eqn]$ nodes, where $[eqn]$ nodes from cluster 1 in two phases with $[eqn]$ , and $[eqn]$ nodes from cluster 2 in two phases with $[eqn]$ .

4. Trade-Off Between the Storage Capacity and Repair Bandwidth

In this section, we derive a bound on the file size when the system parameters are fixed. The bound indicates the trade-off between the storage capacity $[eqn]$ and the repair bandwidth $[eqn]$ .

By inspecting the information flow graph, as described in Section 3, we can obtain the following min-cut bound.

Lemma 1. The min-cut bound between the source S and the data collector $[eqn]$ is given by

$$[eqn]$$

where

$$[eqn]$$

The details of the examination for the cut in Lemma 1 are deferred to Appendix A. Here, we illustrate a possible cut in the information flow graph in Figure 1. The red lines in clusters 1 and 2 give a cut as an example. The contribution to the cut in cluster 1 is $[eqn]$ , and the contribution to the cut in cluster 2 is $[eqn]$ . So, this cut contains a $[eqn]$ amount of information.

According to the minimum cut–maximum flow theorem, we know that the file size M is bounded by the min-cut of the aforementioned information flow graph.

Theorem 1. A file of size M stored in a clustered distributed storage system with multiple failures and zero cross-cluster bandwidth must satisfy

$$[eqn]$$

where $[eqn]$ is the parameter defined in Lemma 1.

An immediate corollary is as follows:

Corollary 1. When $[eqn]$ , we have

$$[eqn]$$

which is the bound that can be derived from Theorem 1 in [13].

An example: The trade-off between the storage capacity and the repair bandwidth for clustered distributed storage systems equipped with the multi-clustered collaborative repair codes (MCCRCs) presented in this article is illustrated in Figure 2, compared with the trade-off for multi-clustered systems with a single failure to repair (i.e., systems equipped with multi-clustered repair codes (MCRCs)) [13]. The parameters in the example are specified as follows: the clustered system has $[eqn]$ nodes in $[eqn]$ clusters, each with a size of $[eqn]$ , and any $[eqn]$ nodes can retrieve the file. $[eqn]$ nodes need to be repaired in each cluster, where 2 nodes are repaired simultaneously for the MCCRC, while 2 nodes are repaired separately and sequentially for the MCRC. An immediate observation from Figure 2 is that the MCCRC employing collaboration further reduces the repair bandwidth compared to that with an MCRC when repairing multiple failures.

5. Two Extreme Points

From Theorem 1, we can see that there are two special cases where the equality holds, namely the minimum storage clustered collaborative repair (MSCCR) point and the minimum bandwidth clustered collaborative repair (MBCCR) point. The values of the parameters in these two special cases are as follows.

The minimum storage clustered collaborative repair (MSCCR) point: Let $[eqn]$ , $[eqn]$ be the floor function; the parameters of the minimum storage clustered collaborative repair (MSCCR) point on the trade-off curve have the following values:

$$[eqn]$$

The minimum bandwidth clustered collaborative repair (MBCCR) point: Let $[eqn]$ , parameters of the minimum bandwidth clustered collaborative repair (MBCCR) point on the trade-off curve have the following values:

$$[eqn]$$

The details on identifying the parameters for the MSCCR point and the MBCCR point can be found in Appendix B.

6. Bandwidth Analysis

To understand the effect of the different values of the parameters on the repair bandwidth, we consider the normalized repair bandwidth $[eqn]$ . We will compute the value of the normalized repair bandwidth for different l values, the number of nodes in a cluster, at the minimum storage clustered collaborative repair (MSCCR) point and the minimum bandwidth clustered collaborative repair (MBCCR) point, respectively. It is shown that the normalized repair bandwidth is not monotonic with respect to l at the MSCCR point or the MBCCR point, respectively. Furthermore, we show that the performance of $[eqn]$ at the MSCCR and MBCCR points is different.

6.1. The MSCCR Point

The normalized repair bandwidth for the minimum storage is

$$[eqn]$$

where i is an integer, and $[eqn]$ .

It is shown that the normalized repair bandwidth $[eqn]$ at the MSCCR point is not monotonic with respect to the cluster size l. Specifically, on the interval $[eqn]$ , $[eqn]$ is increasing with respect to l. On the interval $[eqn]$ , $[eqn]$ is a decreasing function. The normalized repair bandwidth $[eqn]$ is a piecewise monotonic function with respect to l.

In Figure 3, we compare the normalized repair bandwidth at the MSCCR point with different k and t values, where $[eqn]$ . Since $[eqn]$ , we can select $[eqn]$ . In Figure 3a, we compare $[eqn]$ for $[eqn]$ and $[eqn]$ when $[eqn]$ . When t is fixed, a bigger value for k gives a smaller repair bandwidth at the MSCCR point for every $[eqn]$ . In Figure 3b, we compare $[eqn]$ for $[eqn]$ and $[eqn]$ when $[eqn]$ . When k is fixed, a bigger value for t not always gives a bigger $[eqn]$ but gives a smaller $[eqn]$ for the majority of l.

6.2. The MBCCR Point

The normalized repair bandwidth for the minimum storage is

$$[eqn]$$

where i is an integer, and $[eqn]$ .

We found that the normalized repair bandwidth $[eqn]$ at the MBCCR point was not monotonic with respect to the cluster size l. Specifically, on the interval $[eqn]$ , $[eqn]$ is a decreasing function. However, on the interval $[eqn]$ , $[eqn]$ is not a monotonic function with respect to l. Moreover, the normalized repair bandwidth $[eqn]$ at the MBCCR point is not a piecewise monotonic function with respect to l, which is different from the case at the MSCCR point.

In Figure 4, we compare the normalized repair bandwidth at the MBCCR point with different values for k and t, where $[eqn]$ . We select $[eqn]$ . In Figure 4a, we compare $[eqn]$ for $[eqn]$ and $[eqn]$ when $[eqn]$ . When t is fixed, a bigger value for k gives a smaller repair bandwidth at the MBCCR point for every $[eqn]$ . In Figure 4b, we compare $[eqn]$ for $[eqn]$ and $[eqn]$ when $[eqn]$ . When k is fixed, a bigger value for t gives a bigger value for $[eqn]$ for every $[eqn]$ . It is shown that the performance of $[eqn]$ at the MBCCR point with different values of t is different from the case at the MSCCR point.

Moreover, we compare the normalized repair bandwidth at the MSCCR point and the MBCCR point. In Figure 5a, we compare $[eqn]$ at the MSCCR point and the MBCCR point with $[eqn]$ and $[eqn]$ . In Figure 5b, we compare $[eqn]$ at the MSCCR point and the MBCCR point with $[eqn]$ and $[eqn]$ . Obviously, with the same system parameters, $[eqn]$ at the MSCCR point is bigger than that at the MBCCR point. When the number of failures t changes, the lower t is, the greater the difference in $[eqn]$ between the MSCCR point and the MBCCR point. In other words, when k, l is fixed, fewer node failures occur, and the greater the benefit to the repair bandwidth of using MBCCR codes.

7. Code Constructions

In this section, we propose two constructions satisfying the conditions of the MBCCR point and the MSCCR point, respectively, with the parameters $[eqn]$ , $[eqn]$ . We use two layers of codes for encoding; the outer layer uses a maximum rank distance (MRD) code, for example, a Gabidulin code. We will introduce MRD codes and Gabidulin codes first.

Definition 1. An $[eqn]$ rank metric code $[eqn]$ (where q is a prime) is a linear code over $[eqn]$ of the length N, the dimension K, and the maximum rank D [46,47]. A rank metric code that attains a Singleton-like bound $[eqn]$ on the rank metric is called a maximum rank distance (MRD) code.

Here, the rank distance between two vectors $[eqn]$ is defined as $[eqn]$ , where $[eqn]$ is the maximum number of linearly independent coordinates of $[eqn]$ over the base field $[eqn]$ , for a given basis of $[eqn]$ over $[eqn]$ . Codes utilizing this distance metric are referred to as rank metric codes.

Definition 2. An $[eqn]$ rank metric code over the extension field $[eqn]$ achieving the maximum rank distance $[eqn]$ (for $[eqn]$ ) can be constructed using the following linearized polynomial:

$$[eqn]$$

This is referred to as a Gabidulin code. A codeword in a Gabidulin code is defined as

$$[eqn]$$

with the coefficients given by the information message, and $[eqn]$ are linearly independent over $[eqn]$ .

We remark that a linearized polynomial $[eqn]$ satisfies $[eqn]$ for any $[eqn]$ and $[eqn]$ . Given the evaluations of $[eqn]$ at any K linearly independent (over $[eqn]$ ) point in $[eqn]$ , one can recover $[eqn]$ and therefore reconstruct the message vector $[eqn]$ by performing polynomial interpolation (called the MDS property of Gabidulin code in the following content). These properties will be utilized in our code constructions.

7.1. Minimum Bandwidth Clustered Collaborative Repair (MBCCR) Codes

Consider a file of size $[eqn]$ (the file can be seen as a data object vector of the length M), where $[eqn]$ , with coefficients in the finite field $[eqn]$ and q being a prime. Encoding the object using an $[eqn]$ Gabidulin code, $[eqn]$ and $[eqn]$ . The codeword $[eqn]$ is first divided into L groups denoted by $[eqn]$ , $[eqn]$ , with each being $[eqn]$ in size.

Rewriting each $[eqn]$ , $[eqn]$ as the matrix form,

$$[eqn]$$

where $[eqn]$ is an $[eqn]$ matrix. For each $[eqn]$ , let $[eqn]$ be an $[eqn]$ matrix with the entries in $[eqn]$ satisfying the principles that any $[eqn]$ columns of $[eqn]$ are linearly independent, $[eqn]$ is an invertible $[eqn]$ matrix with the entries in $[eqn]$ , and any $[eqn]$ sub-matrix of $[eqn]$ is invertible. $[eqn]$ and $[eqn]$ can be constructed using a Vandermonde matrix.

Data placement in each cluster: In each cluster r, $[eqn]$ , node i, $[eqn]$ stores $[eqn]$ and $[eqn]$ . Since $[eqn]$ , node i stores $[eqn]$ symbols.

The repair of t failures in each cluster: In each cluster r, $[eqn]$ , without a loss of generality, suppose that the first t nodes fail. Each failed node i, $[eqn]$ downloads $[eqn]$ and $[eqn]$ from the nodes that are still alive j, $[eqn]$ . Since any $[eqn]$ columns of $[eqn]$ are linearly independent, the node i obtains $[eqn]$ ; furthermore, it can compute $[eqn]$ . The total number of symbols downloaded from the helper nodes to one new node in the download phase of the repair process is $[eqn]$ .

The missing $[eqn]$ can be obtained by contacting the other $[eqn]$ nodes being repaired. Node i contacts nodes $[eqn]$ and downloads $[eqn]$ from each, and then node i obtains $[eqn]$ . Since $[eqn]$ is invertible, node i can obtain $[eqn]$ . The total number of symbols downloaded during the collaboration process is $[eqn]$ , that is, $[eqn]$ ; thus, $[eqn]$ .

Data retrieval: When any k nodes are connected by a $[eqn]$ , at least $[eqn]$ $[eqn]$ linearly independent symbols over $[eqn]$ in $[eqn]$ can be obtained, where $[eqn]$ . Indeed, in each cluster r, although the storage capacity is $[eqn]$ , each cluster only contains $[eqn]$ linearly independent symbols since only $[eqn]$ linearly independent symbols in $[eqn]$ are coded. For the k nodes being contacted, the fewer clusters they are in, the fewer independent symbols can be obtained. When $[eqn]$ , at least $[eqn]$ clusters are being contacted, and we denote these clusters as cluster $[eqn]$ . We find that when all the nodes of $[eqn]$ (out of $[eqn]$ ) clusters are being contacted by the $[eqn]$ , the $[eqn]$ will obtain the fewest linearly independent symbols. Therefore, without a loss of generality, suppose that all of the nodes in clusters $[eqn]$ are being contacted; then, $[eqn]$ linearly independent symbols are obtained from cluster $[eqn]$ . In the cluster $[eqn]$ , only $[eqn]$ nodes are being contacted. Here, we notice that in each cluster, $[eqn]$ nodes are able to repair t failures; thus, at most $[eqn]$ $[eqn]$ symbols are linearly independent in each cluster. So, from $[eqn]$ nodes, only $[eqn]$ linear independent symbols are obtained since the construction has the property $[eqn]$ for $[eqn]$ in the same cluster r. Therefore, from any k nodes of the system, at least $[eqn]$ $[eqn]$ linearly independent symbols in $[eqn]$ are obtained, and they are all the codeword symbols of the outer Gabidulin code. Accordign to the MDS property of Gabidulin codes, any $[eqn]$ symbols are linearly independent and are able to retrieve the data object. Therefore, from any k nodes, the $[eqn]$ can obtain at least $[eqn]$ linearly independent symbols so that the file is retrieved.

The parameters of the construction are $[eqn]$ , $[eqn]$ , which satisfy the condition at the MBCCR point when $[eqn]$ $[eqn]$ .

In the MBCCR construction, we consider the information symbols over the finite field $[eqn]$ . There are two layers of codes in the constructions. The outer code is an $[eqn]$ Gabidulin code over $[eqn]$ , with $[eqn]$ and $[eqn]$ . The inner code is a product-matrix code over $[eqn]$ . The node size is mainly determined by the outer Gabidulin code. Each symbol in $[eqn]$ can be seen as the vector over the field $[eqn]$ of the length $[eqn]$ , without a loss of generality. Let $[eqn]$ ; so, the node size for the MBCCR code is $[eqn]$ . Furthermore, $[eqn]$ scales as $[eqn]$ , if we fix the number of clusters L to a relatively small number compared with the number of nodes n.

7.2. Minimum Storage Clustered Collaborative Repair (MSCCR) Codes

Consider a data object of size $[eqn]$ with coefficients in the finite field $[eqn]$ , where $[eqn]$ . Encoding the object of size M by an $[eqn]$ Gabidulin code, with $[eqn]$ . The codeword $[eqn]$ is first divided into L groups, denoted by $[eqn]$ , $[eqn]$ , with each being $[eqn]$ in size.

Rewriting each $[eqn]$ , $[eqn]$ as a $[eqn]$ matrix $[eqn]$ ,

$$[eqn]$$

where $[eqn]$ is a row vector of size $[eqn]$ . For each $[eqn]$ , let $[eqn]$ be a generator matrix of an $[eqn]$ MDS code over $[eqn]$ , where $[eqn]$ , for $[eqn]$ .

Data placement in each cluster: In each cluster r, $[eqn]$ , node i stores $[eqn]$ symbols $[eqn]$ , where

$$[eqn]$$

Repair of t failures in each cluster: In each cluster r, $[eqn]$ , when t nodes fail, without a loss of generality, we label t nodes from 1 to t. Then, these t nodes each contact the $[eqn]$ nodes that are still alive; say the i-th node among these t failed nodes connects to nodes $[eqn]$ and downloads $[eqn]$ from the alive nodes $[eqn]$ , $[eqn]$ . Using the MDS property of $[eqn]$ , node i can obtain $[eqn]$ and compute $[eqn]$ . The total number of symbols downloaded from the helper nodes to one node in the download phase of the repair process is $[eqn]$ .

The ith node contacts nodes $[eqn]$ , $[eqn]$ , and downloads $[eqn]$ to obtain the missing $[eqn]$ symbols. The total number of symbols downloaded during the collaboration process is $[eqn]$ , that is $[eqn]$ ; thus, $[eqn]$ .

Data retrieval: When any k nodes are connected by a $[eqn]$ , at least $[eqn]$ $[eqn]$ linearly independent symbols (over $[eqn]$ ) in $[eqn]$ are obtained, where $[eqn]$ . Indeed, in each cluster r, only $[eqn]$ linearly independent symbols in $[eqn]$ are coded and stored. For the k nodes being contacted, the fewer clusters they are in, the fewer linear independent symbols are obtained. When $[eqn]$ , at least $[eqn]$ clusters are contacted, and we denote these clusters as cluster $[eqn]$ . Without a loss of generality, suppose that all of the nodes in clusters $[eqn]$ are being contacted; then, $[eqn]$ linearly independent symbols are obtained. In the cluster $[eqn]$ , only $[eqn]$ nodes are being contacted, and then $[eqn]$ linear independent symbols are obtained. Therefore, from any k nodes, at least $[eqn]$ linearly independent symbols are obtained, and they are all of the codeword symbols of the outer Gabidulin code. According to the MDS property of Gabidulin codes, any $[eqn]$ symbols are linearly independent and are able to retrieve the object. Therefore, from any k nodes, the $[eqn]$ can obtain at least $[eqn]$ linearly independent symbols so that the file is retrieved.

The parameters of the construction are $[eqn]$ , $[eqn]$ , which satisfy the condition of the MSCCR point when $[eqn]$ .

In the MSCCR code construction, the outer code is a Gabidulin code over $[eqn]$ with $[eqn]$ , and the inner code is an $[eqn]$ -MDS code over $[eqn]$ . Similarly to the discussion for MBCCR codes, we know that the node size for MSCCR code is $[eqn]$ . Moreover, $[eqn]$ can be scaled as $[eqn]$ if L is a relatively small number compared with n.

We give a simple example of an MSCCR code below to illustrate the construction.

Example 1. Consider a system with its parameters as shown in Figure 1. The parameters have the values $[eqn]$ , $[eqn]$ , $[eqn]$ .

We encode the object using an $[eqn]$ Gabidulin code. The obtained codeword $[eqn]$ is first divided into two groups denoted by $[eqn]$ , $[eqn]$ , with each being 6 in size.

Rewriting each $[eqn]$ ( $[eqn]$ ), as a $[eqn]$ matrix $[eqn]$ , that $[eqn]$ . For each $[eqn]$ , let $[eqn]$ be a generator matrix of an $[eqn]$ MDS code over $[eqn]$ , where $[eqn]$ , for $[eqn]$ . In each cluster $[eqn]$ , node i stores $[eqn]$ symbols $[eqn]$ and $[eqn]$ . Repair: *In each cluster $[eqn]$ , each failed node i, $[eqn]$ downloads $[eqn]$ from the nodes that are alive $[eqn]$ , $[eqn]$ . Using the MDS property of $[eqn]$ , the i-th failed node can obtain $[eqn]$ and compute $[eqn]$ . The total number of symbols downloaded from the helper nodes to one node in the download phase of the repair process is $[eqn]$ . Then, the ith node contacts nodes $[eqn]$ and downloads $[eqn]$ to obtain the missing 1 symbols. The total number of symbols downloaded during the collaboration process is 2, that is, $[eqn]$ ; thus, $[eqn]$ .*Data retrieval: When any six nodes are connected by a data collector, at least eight linearly independent symbols (over $[eqn]$ ) in $[eqn]$ are obtained. Indeed, in each cluster $[eqn]$ , only six symbols in $[eqn]$ are linearly independent. When six nodes are being contacted, with five from the first cluster and one from the second cluster, the $[eqn]$ will obtain the fewest linearly independent symbols. Specifically, six linearly independent symbols in $[eqn]$ are obtained from the first cluster, and from the second cluster, only two linear independent symbols are obtained, totalling eight linearly independent symbols over $[eqn]$ that are obtained, and they are all the codeword symbols of the outer Gabidulin code. According to the MDS property of Gabidulin codes, any eight linearly independent symbols over $[eqn]$ in $[eqn]$ are able to retrieve the object. Therefore, from any six nodes, the data collector can obtain at least eight linearly independent symbols, and then the file is retrieved.Figure 6 illustrates a possible scenario where in each cluster, $[eqn]$ nodes fail. In cluster $[eqn]$ , nodes 2 and 4 fail. The failed node 2 can be seen as the first failed node, and then it downloads $[eqn]$ from the nodes that are still alive $[eqn]$ . The failed node 4 can be seen as the second failed node and downloads $[eqn]$ from the alive nodes $[eqn]$ . When $[eqn]$ nodes are connected, 5 nodes from cluster 1 and 1 node from cluster 2, we obtain 12 symbols in $[eqn]$ . For example, in the second cluster, the fourth node is contacted, and then we obtain twelve symbols $[eqn]$ in $[eqn]$ , and eight of them are linearly independent over $[eqn]$ ; then, according t the MDS property of the outer $[eqn]$ Gabidulin code, the file can be retrieved.

8. Conclusions

We have studied the repairing problem in a clustered distributed storage system, where each cluster is resilient to multiple failures within itself. A fundamental trade-off between the storage and repair bandwidth is derived by analyzing the associated information flow graph. Two extreme points on the trade-off curve, i.e., the MSCCR point and the MBCCR point, are studied. Their performance in terms of the repair bandwidth is studied as well. Furthermore, explicit and optimal code constructions for the two points are presented. We considered an ideal situation where the cross-cluster bandwidth is zero in this article. However, in a real clustered system, cross-cluster communication is possible, as stated in [8,9,10]. Therefore, a future direction for our study is to incorporate cross-cluster communication into the system. Moreover, since we proposed using Gabidulin codes as the outer codes for the optimal code constructions, which resulted in a larger node size, using other optimal or sub-optimal code constructions with a smaller node size would be another future direction.