The Storage vs Repair-Bandwidth Trade-off for Clustered Storage Systems

TL;DR

This paper explores the fundamental trade-offs in clustered storage systems between storage efficiency and repair bandwidth, proposing optimal codes and bounds for both functional and exact repairs, considering intra- and inter-cluster communication costs.

Contribution

It introduces a generalized model for clustered storage, derives the optimal trade-off between storage and repair bandwidth, and provides new code constructions and bounds for intra- and inter-cluster repairs.

Findings

Optimal trade-off between storage and inter-cluster repair bandwidth identified.

New exact-repair code constructions outperform space-sharing solutions.

Bounds on intra-cluster repair bandwidth reveal increased helper nodes impact.

Abstract

We study a generalization of the setting of regenerating codes, motivated by applications to storage systems consisting of clusters of storage nodes. There are $n$ clusters in total, with $m$ nodes per cluster. A data file is coded and stored across the $mn$ nodes, with each node storing $\alpha$ symbols. For availability of data, we require that the file be retrievable by downloading the entire content from any subset of $k$ clusters. Nodes represent entities that can fail. We distinguish between intra-cluster and inter-cluster bandwidth (BW) costs during node repair. Node-repair in a cluster is accomplished by downloading $\beta$ symbols each from any set of $d$ other clusters, dubbed remote helper clusters, and also up to $\alpha$ symbols each from any set of $\ell$ surviving nodes, dubbed local helper nodes, in the host cluster. We first identify the optimal trade-off between storage-overhead and inter-cluster repair-bandwidth under functional repair, and also present optimal exact-repair code constructions for a class of parameters. The new trade-off is strictly better than what is achievable via space-sharing existing coding solutions, whenever $\ell > 0$. We then obtain sharp lower bounds on the necessary intra-cluster repair BW to achieve optimal trade-off. Our bounds reveal the interesting fact that, while it is beneficial to increase the number of local helper nodes $\ell$ in order to improve the storage-vs-inter-cluster-repair-BW trade-off, increasing $\ell$ not only increases intra-cluster BW in the host-cluster, but also increases the intra-cluster BW in the remote helper clusters. We also analyze resilience of the clustered storage system against passive eavesdropping by providing file-size bounds and optimal code constructions.

Figures

Tables

Table 1. TABLE I: Notation used in our System Model. The last three parameters relate to intra-cluster repair-bandwidth

Symbol	Definition
$n$	total number of clusters in the system
$m$	number of storage nodes in each cluster
$k$	number of clusters required for data collection
$d$	number of remote helper clusters providing helper data during node repair
$\ell$	number of local helper nodes providing helper data during node repair
$q$	finite field size for data symbols
$\alpha$	number of symbols each storage node holds for one coded file
$\beta$	size of helper data downloaded from each remote helper cluster during node repair, in symbols
\hdashline$\gamma$	size of helper data downloaded from each local helper during node repair, in symbols
${\ell }^{\prime }$	number of nodes in a remote helper cluster that contribute toward computing the $\beta$ helper symbols of the cluster
${\gamma }^{\prime }$	size of the data provided by each of ${\ell }^{\prime }$ nodes at remote helper cluster to compute the cluster helper data, in symbols

Table 2. TABLE II: Summary of related works, and key differences from generalized regenerating codes (GRC)

Reference	Focus
RCs in clustered topologies
Hu et al. [15], Sohn et al. [17], Gastón et al. [18]	Rack-based topology with cluster-based repair, allow use of cheaper local-rack helper data. [15] and [17] permit pooling of intra-rack helper data to decrease inter-rack bandwidth. Unlike GRC, the works assume node-based data collection with no notion of clustering for data collection
Pernas et al. [19]	Similar to Gastón et al. [18] with 2 racks with different per-node storage
Calis and Koyluoglu [20]	Two-layered storage with nodes grouped in blocks, block failure model, repair and data collection from fixed number of nodes from surviving blocks. All links have same costs
Rashmi et al. [21]	Two sets of nodes, repair from $d$ nodes of the other set, data collection from $k$ nodes of any single set. All links have same costs
RCs in heterogeneous systems
Shah et al. [22]	Non-uniform amount of data downloaded from nodes during repair or data collection, without any notion of clustering
Yu et al. [23], Ernvall et al. [24]	Non-uniform storage sizes and repair BW costs
Akhlaghi et al. [25]	Repair BW cost can take two different values, depending on the helper node used, this is different from clustering
Li et al. [26], Wang et al. [27]	Aim to transmit the repair helper data in shortest amount of time by finding the optimal path from helper nodes to replacement nodes through a network with non-uniform link capacities
Gopalan et al. [14]	Inherent notion of clustering for node repair, and single node is performed locally with the help of other nodes in the clusters. Thus unlike GRC, LRCs assume dependency among nodes in a cluster for single node repair. Further, there is no notion of clustering for data collection.

Equations

$$B$$

$$\gamma$$

$${\gamma }^{\prime }$$

$$B^{(s)}$$

$$B$$

$$B_1^{\ast }$$

$$B_m^{\ast }$$

$$B$$

$$\le$$

$$=$$

$$B$$

$$\text{mincut}(S-T)$$

$$=$$

$$\ge$$

$$A_{m\times m}$$

$$[{\mathbf{c}}_1^{\prime }{\mathbf{c}}_2^{\prime }\cdots {\mathbf{c}}_m^{\prime }]$$

$$=$$

$$A_{m\times m}$$

$$=$$

$$\displaystyle\left[{\bf c}^{(i)}_{MDS}\ \ {\bf c}^{(i)}_{regen}\right]\left[\begin{array}[]{c}{\bf e}_{j}^{T}\\ \hline\cr\\ {\bf f}_{j}^{T}\end{array}\right]$$

$$\displaystyle\left(\left[{\bf c}^{(1)}_{MDS}\ \ {\bf c}^{(1)}_{regen}\right]\left[\begin{array}[]{c}E^{\prime}\\ \hline\cr\\ F^{\prime}\end{array}\right]\right)E^{\prime-1}{\bf e}_{\ell+1}^{T}$$

$$Y_{i^{\prime },j^{\prime }}(t)$$

$$Y_{i,j}(t)=Y_{i,j}(t-1).$$

$$\sum _{j=1}^m{Y}_{i^{\prime },j}(t^{\prime }+1)$$

$$\sum _{j=1}^m{Y}_{i,j}(t^{\prime }+1)$$

$$\gamma$$

$$C_{cut}$$

$$=k\ell \alpha +(m-\ell )\sum _{i=0}^{k-1}\min \{\alpha ,(d-i{)}^{+}\beta \}-\alpha +\min \{\alpha ,(d-k+1{)}^{+}\beta \}+\gamma$$

$$=B^{\ast }-\alpha +\min \{\alpha ,(d-k+1{)}^{+}\beta \}+\gamma .$$

$$\text{mincut}(S-T)$$

$$(\ell -j+1{)}^{+}\gamma +(d-(i-1){)}^{+}\beta$$

$${\gamma }^{\prime }$$

$$C_{cut}$$

Full text

I Introduction

We consider the problem of designing efficient erasure codes for fault-tolerant data storage in a clustered network of storage nodes. Nodes within a cluster are connected to each other via one network, while a second network provides connectivity between clusters. In clustered networks, there is often a differentiation between intra- and inter-cluster bandwidth costs, and this occurs because of factors like physical distance between the clusters or some other differentiating characteristic in the communications within and between clusters. Typically, intra-cluster bandwidth cost is much less than inter-cluster bandwidth cost. A user file is erasure coded and stored across the nodes in the various clusters. From an availability perspective, it is of interest to encode and store a data file such that access to a subset of clusters allows reconstruction of the entire uncoded file. We consider nodes as failure domains, and require efficient repair of failed nodes in any cluster. Repair of a failed node in a cluster is performed by downloading helper data from other clusters, and also a subset of surviving nodes in the host-cluster. We permit the possibility that a cluster that aids in the repair of a failed node gathers and processes helper-data from its various nodes, before sending it to the target-node, in order to decrease inter-cluster repair-bandwidth costs. A good erasure-code solution allows a desirable trade-off between storage-overhead and the inter- and intra-cluster repair BW costs for a given availability requirement.

The model described above is motivated by applications to cloud storage settings, where user data is spread across distinct data-centers. For instance, the clusters could represent geographically separated data centers of a cloud-service provider or a content-delivery network. All major cloud service providers like Amazon [1], Microsoft Azure [2] and Akamai [3] provide options for geo-replicating user data in multiple data centers. Geo-replication across data centers improves both availability and durability of user data. Data-center unavailability occurs owing to network or power outages, software bugs and even security vulnerabilities. Note that in the event of unavailability, data is never lost and thus does not necessitate data-center wide rebuild. While the three systems mentioned above do not mention erasure coding across data centers, other practical systems indeed consider this possibility. For example, Facebook’s F$4$ blob storage system [4] considers storing the xored content of two data centers in a third data-center. Similarly, the authors of the Hitachi white-paper [5] suggest using Reed-Solomon like codes across data centers for private clouds consisting of a small number of data-centers. Microsoft Giza [6], a geo-distributed storage system considers erasure coding data across as many as $11$ data centers spread over $3$ continents. Giza employs MDS codes like Reed-Solomon codes across data centers to handle availability requirement during data collection; i.e., a user connects to the nearest $k$ available data centers (assuming an $[n,k]$ MDS code is used) during data collection to download the data. The paper notes that (we quote) “cross-DC erasure coding only becomes economically attractive if 1) there are workloads that consume very large storage capacity while incurring very little cross-DC traffic; 2) there are enough cross-DC network bandwidth at very low cost”. The paper provides justification for both these aspects; we refer to [6] for details.

Our model is perhaps even more applicable to the setting of “cloud of clouds”, where user data is spread across data centers corresponding to multiple cloud service providers. Implementation studies that show the advantages of using Reed-Solomon-like erasure codes to store data in user-defined cloud-of-clouds appear in [7, 8, 9, 10]. Several reasons motivate a cloud-of-cloud setting, instead of a single cloud setting. The first is the need for high availability in a local geographic zone [8] - individual cloud service providers might be limited in their number of geographic zone specific data centers. Another motivation for a cloud-of-cloud setting is the flexibility to avoid vendor lock-in [7, 8]. With an ever increasing number of new cloud providers offering competitive pricing and features, it might be of interest to migrate from an existing cloud provider to a new one. However, individual cloud providers charge the users differently for in-network (intra-cluster in our model) and out-of-network (inter-cluster) data movement. In this scenario, to decrease the cost of migration, it is beneficial to spread data across several providers, so that the user only needs to migrate data in the less competitive providers to the new provider. Finally, a reason for providing user-defined cloud-of-clouds is that of privacy concerns [9, 10, 11], where security compromise of any one cloud provider does not compromise user data.

In this work, we model an entire data-center as a cluster. We restrict ourselves to the storage of a single user file; the solution provided here can be applied independently to any of the user files. For storage of a single coded file, we assume an equal number of storage nodes in any of the clusters. While performing data collection, we assume a cluster to be either completely available or completely unavailable. Such an assumption is sensible in a multi-data-center cloud setting [5]. In our model, we restrict ourselves to recovery from single node failure. If there are multiple node failures in the system, it is assumed that the recovery process happens sequentially, one node at a time. While catastrophic failure of an entire data-center is rare, correlated failures of nodes in a data center are an important issue reported in practice [12]. In this scenario, in our model, we parametrize the number of nodes, named local helper nodes, in the host-cluster that can aid in the repair of a node in the host-cluster. Further, to keep the model simple, we ignore any hierarchical topologies that may be present inside a data-center (cluster), and simply assume equal cost connectivity between any two nodes inside a cluster. We also assume direct connectivity between any two clusters in the network.

A straightforward solution that guarantees availability of data, and minimizes inter-cluster repair-bandwidth costs is simply to use a product code consisting of two Maximum Distance Separable (MDS) codes - one across the clusters, and another within a cluster. While the solution entirely eliminates any inter-cluster bandwidth cost, it suffers from poor storage-overhead owing to the need to have redundancy in every cluster. At the other extreme, it is also possible to achieve highly optimized storage cost, at the expense of inter-cluster repair-bandwidth, by using minimum storage regenerating (MSR) codes [13] across the clusters. For instance, if we assume there are $n$ clusters in total, then $(n,k)$ MSR codes across the clusters ensure that data is retrievable by accessing (entire) content of any $k$ clusters. However, the solution suffers from high inter-cluster repair-bandwidth cost. Our goal in this work, at a high level, is to explore alternate solutions which can smoothly trade-off storage-overhead against inter-cluster repair-bandwidth for given availability requirements. We show that it is indeed possible to achieve (see Fig. 6) operating points which are strictly better than those obtained by space-sharing between product MDS codes and MSR codes. We also characterize the amount of intra-cluster repair-bandwidth needed for achieving the optimal trade-off between storage-overhead and inter-cluster repair bandwidth.

For the rest of the introduction, we first provide an abstract description of the system model used in this work. This is followed by a discussion of other related system models in the literature, a summary of our results, and also an example of the proposed erasure-coding solution.

I-A System Model

We propose a natural generalization of the setting of regenerating codes (RC) [13] for clustered storage networks. The network consists of $n$ clusters, with $m$ nodes in each cluster. The network is fully connected such that any two nodes within a cluster are connected via an intra-cluster link, and any two clusters are connected via an inter-cluster link. A node in one cluster that needs to communicate with another node in a second cluster does so via the corresponding inter-cluster link. A data file of size $B$ symbols is encoded into $nm\alpha$ symbols, and stored across the $nm$ nodes such that each node stores $\alpha$ symbols. The symbols are assumed to come from a finite field $\mathbb{F}_{q}$ of $q$ elements. For data collection we have an availability constraint such that the entire content of any $k$ clusters be sufficient to recover the original data file (Fig. 1). As mentioned before, nodes represent failure domains, and we restrict ourselves to the case of efficient recovery from single node failure. Node repair is parametrized by three parameters $d,\beta$ and $\ell$. We assume that the replacement of a failed node is in the same cluster as the failed node. The replacement node downloads $\beta$ symbols each from any set of $d$ other clusters, dubbed remote helper clusters. The $\beta$ symbols from any of the remote helper clusters are possibly a function of the $m\alpha$ symbols present in the cluster - we assume that any one of the nodes in the cluster takes responsibility for computing these $\beta$ symbols before passing them outside the cluster111One could also assume the presence of a dedicated compute unit to compute the helper data. Such an assumption allows us to enforce a symmetric demand on the usage of intra-cluster bandwidth for all the nodes in the remote helper cluster. We will rely on the presence of such dedicated compute units in the information flow graphs (see Section III) used to derive file-size upper bounds.. Further, we also permit the replaced node to download (entire) content from any set of $\ell$ other nodes, dubbed local helper nodes, in the host cluster, during the repair process. The quantity $d\beta$ represents the inter-cluster repair-bandwidth. We refer to the overall code as the generalized regenerating code (GRC) $\mathcal{C}_{m}$ with parameters $\{(n,k,d)(\alpha,\beta)(m,\ell)\}$.

The model reduces to the setup of RCs in [13], when $m=1$ (in which case, $\ell=0$ automatically). We shall refer to the setup in [13] as the classical setup or classical regenerating codes. Our generalization has two additional parameters $\ell$ and $m$ when compared with the classical setup. As in the classical setup we consider both the notions of functional and exact repair. Under exact repair, the content of the repaired node is identical to that of the failed node; while in functional repair, the repair content permits data collection and repair of additional failed nodes. The first goal of the paper is to obtain a trade-off between storage-overhead $nm\alpha/B$ and inter-cluster repair-bandwidth $d\beta$ for an $\{(n,k,d)(\alpha,\beta)(m,\ell)\}$ GRC. We further note that, unlike the classical setup, the generalized setup permits $d<k$. In our model, whenever $d>0$, we assume that the encoding function does not introduce any local dependence among the content of the various nodes of a cluster222Under linear encoding, the coded content of cluster $i,1\leq i\leq n$ can be written as $\widehat{\bf m}G_{i}$, where $\widehat{\bf m}$ is the message vector of length $B$, and $G_{i}$ is a $B\times m\alpha$ matrix. In this case, when we say that the encoding not introduce local dependence, we mean that the matrix $G_{i}$ has full column-rank; for example, the model excludes the possibility of a local parity node within a cluster, which would hold the component-wise sum in $\mathbb{F}_{q}^{\alpha}$ of the other nodes’ data. As we shall show, the case $d=0$ is a special one where local dependence is necessary.

The model described above does not consider intra-cluster bandwidth incurred during repair. Intra-cluster bandwidth is needed, firstly, to compute the $\beta$ symbols in any remote helper cluster, and, secondly, to download content from $\ell$ local helper nodes in the host cluster. In order to characterize the amount of intra-cluster bandwidth that is needed to establish optimal trade-off between storage-overhead and inter-cluster repair-bandwidth, we consider the repair model shown in Fig. 1(b). In this model, the replacement node downloads at most $\gamma,\gamma\leq\alpha$ symbols from each of the $\ell$ local helper nodes from the host-cluster. With regard to a remote helper cluster, we assume that the $\beta$ symbols contributed by it are only a function of at most $\ell^{\prime},\ell^{\prime}\leq m$ nodes of the cluster. We make the assumption that any set of $\ell^{\prime}$ nodes can be used to compute the $\beta$ symbols. Further, we limit the amount of data that each of these $\ell^{\prime}$ nodes can contribute to at most $\gamma^{\prime},\gamma^{\prime}\leq\alpha$ symbols. A second goal of this paper is to identify necessary requirements on the parameters $\gamma,\ell^{\prime},\gamma^{\prime}$ that are needed to guarantee optimal trade-off between storage-overhead and inter-cluster repair-bandwidth.

A summary of the various parameters used in the description of the system model appears in Table I.

I-B Related Work

Regenerating codes were originally introduced in [13] for simultaneously optimizing storage overhead and repair bandwidth for flat storage systems. By a flat storage system, it is meant that every node in the storage system is connected to every other storage node via some logical link, where all logical links incur the same bandwidth cost for communication per bit. Further, the data collector also connects to any of the storage nodes via links of similar cost. There has since then been significant progress in the area of classical RCs in terms of code constructions, finding optimal trade-offs under exact repair, and practical implementation. Below, we review variations of RCs that have been proposed for non-flat topologies, and see how our model relates to these existing variations. We shall also comment on how the model of locally repairable codes [14] relates to our model.

I-B1 Regenerating Code Variations for Clustered Topologies

Regenerating code variations for data-center like topologies consisting of racks and nodes are considered in [15, 16, 17, 18, 19, 20]. In [15], [17] and [18], the authors distinguish between inter-rack (inter-cluster) and intra-rack (intra-cluster) bandwidth costs. Further, the works [15] and [17] permit pooling of intra-rack helper data to decrease inter-rack bandwidth. Also, all three works allow taking help from host-rack nodes during repair. Unlike our model, for data collection all three works simply require file retrievability from any set of $k$ nodes irrespective of the racks (clusters) to which they belong. In other words, the notion of clustering applies only to repair, and not data collection, and this is a major difference with respect to our model. Thus while these variations are suitable for modeling the node-rack topologies present within a data center, they do not model situation of erasure coding across data centers with the availability requirement as considered in this work. The work [16] applies the theoretical results of [15] for the practical setting of Hadoop file system. The work in [19] is a variation of that in [18] for a two-rack model, where the per-node storage capacity of the two racks differ. In [20], the authors consider a two-layer storage setting like ours, consisting of several blocks (analogous to clusters as considered in this work) of storage nodes. A different clustering approach is followed for both data collection and node repair. For data collection, one accesses $k_{c}$ nodes each from any of $b_{c}$ blocks. Though [20] focuses on node repair, the model assumes possible unavailability of the whole block where the failed node resides, and as such uses only nodes from other blocks for repair. Further, unlike our model in this work, the authors do not differentiate between inter-block and intra-block bandwidth costs. The framework of twin-codes introduced in [21] is also related to our model and implicitly contains the notion of clustering. In [21] nodes are divided into two sets. For data collection, one connects to any $k$ nodes in the same set. Recovery of a failed node in one set is accomplished by connecting to $d$ nodes in the other set. However, there is no distinction between intra-set and inter-set bandwidth costs, and this becomes the main difference with our model.

I-B2 Regenerating Code Variations for Heterogeneous Systems

Several works [22, 23, 24, 25, 26, 27] study variations of RCs in varied settings, with different combinations of node capacities, link costs, and amount of data-download-per-node. The main difference between our model and these works is that none of them explicitly considers clustering of nodes while performing data collection. In [22], the authors introduce flexible regenerating codes for a flat topology of storage nodes, where uniformity of download is enforced neither during data collection, nor during node-repair. References [23], [24] consider systems where the storage and repair-download costs are non-uniform across the various nodes. The authors of [23], as in [22], allow a replacement node to download an arbitrary amount of data from each helper node. In [25], nodes are divided into two sets, based on the cost incurred while these nodes aid during repair. As noted in [19], the repair model of [25] is different from a clustered network, where the repair cost incurred by a specific helper node depends on which cluster the replacement node belongs to. The works of [26] and [27] focus on minimizing regeneration time rather regeneration bandwidth in systems with non-uniform point-to-point link capacities. Essentially, each helper node is expected to find the optimal path, perhaps via other intermediate nodes, to the replacement node such that the various link capacities are used in a way to transfer all the helper data needed for repair in the shortest possible time. It is interesting to note both of these works permit pooling of data at an intermediate node, which gathers and processes any relayed data with its own helper data. Recall that our model (and the one in [15]) also considers pooling of data within a remote helper cluster, before passing on to the target cluster.

I-B3 Locally Repairable Codes

Locally repairable codes (LRCs), introduced in [14], are motivated by the need to carry out efficient node repair in hierarchical storage systems, such as data centers. The subject of LRCs, like regenerating codes, has attained significant attention both in theory and practice, since its introduction. Recovery of a node failure within a cluster is first attempted locally, in order to minimize cross-cluster bandwidth; if too many nodes in the cluster are unavailable, global parity nodes, spread across various clusters, aid in recovery. However, LRCs do not model clustering of nodes while carrying out data collection, and this is once again a major difference with our model. To draw further similarities with LRCs, we note that in our model, during the repair of a node, the helper data from the $\ell$ nodes from the same cluster can be considered as local-helper data. However if $d>0$, unlike in LRCs, local-helper data alone is not sufficient for single-node-repair, since in this work we assume the encoding function of the generalized regenerating code to introduce no dependency among the content of the $m$ nodes in any cluster. For $d>0$, lack of dependencies within each cluster reduces storage overhead when compared to LRCs; at the same time, local-helper data allows us to achieve a trade-off between storage-overhead and the inter-cluster-repair-bandwidth better than that of a system with $m$ stacked classical RCs, in which the set of $i$-th nodes in all clusters stores the data encoded with $i$-th RC. The case $d=0$ corresponds to one having no remote helper clusters, so that the repair is carried out entirely locally. The difference in this case of $d=0$ with model of LRCs is notion of clustering of nodes, while performing data collection.

A tabular summary of related works appears in Table II.

I-C Our Results

I-C1 Upper Bound on File Size $B$

Under the setting of functional repair, the file-size $B$ is shown to be upper bounded by

[TABLE]

where we use the notation $a^{+}$ to mean $\max(a,0)$, for any integer $a$. The bound is shown by considering the information-flow graph (see Section III) under functional-repair, and calculating the minimum cut. For any finite information-flow graph, the achievability of (1) follows from results in network coding [28]. This establishes the optimality of (1) when we know an upper bound on the number of node failures that occur during the life-time of the system. In practice, random linear network codes (RLNCs) [29] can be used to achieve near-optimal operating points.

For fixed values of $B=B^{*},n,k,d>0,\ell,m$, (1) gives a normalized trade-off (see Fig. 2) between storage-overhead $nm\alpha/B$ and inter-cluster-repair-bandwidth-overhead $d\beta/\alpha$ (see Section IV-A as to why we refer to $d\beta/\alpha$ as repair-bandwidth-overhead). For any $m$, when $\ell=0$, the trade-off is exactly same as that of the classical regenerating codes [13]. When $\ell>0$ (implies $m>1$), the trade-off is strictly better than that of the classical setup.

I-C2 Optimal Code Constructions

We present optimal codes for the minimum storage-overhead and the minimum inter-cluster repair-bandwidth-overhead operating points of the trade-off, for a class of parameters and under the setting of exact repair. These two operating points corresponding to MSR and MBR codes, respectively. The operating points of the MSR and MBR codes are characterized by relations $B=mk\alpha$ and $\alpha=d\beta$ respectively. The construction works by suitably combining $\ell$ $[n,k]$ vector MDS codes over $\mathbb{F}_{q}^{\alpha}$ and $(m-\ell)$ classical exact-repair MSR (MBR) codes. We also present an optimal code construction for functional-repair, which tolerates an arbitrary number of failures333The network-coding based achievability works only if there is a known upper bound on the number of repairs that occur over the duration of operation of the system., for the case $\ell=m-1,d\geq k$. The code is constructed by combining $m-1$ $[n,k]$ vector MDS codes over $\mathbb{F}_{q}^{\alpha}$, and an $\{(n,k,d),(\alpha,\beta)\}$ functional-repair code from [30] for the classical setting. The code construction in [30] is an instance of a functional-repair code for the classical setting, which tolerates an arbitrary number of failures and repairs for the duration of operation of the system.

I-C3 Lower bound on Intra-Cluster Bandwidth Related Parameters

We calculate lower bounds on the intra-cluster-bandwidth related parameters $\gamma,\ell^{\prime},\gamma^{\prime}$, shown in Fig. 1, under assumption that (1) is achieved with equality. While studying the impact of any one of these parameters, we ignore the effects of the other two; for example, the lower bound on $\gamma$ is obtained under the assumption that $\ell^{\prime}=m$ and $\gamma^{\prime}=\alpha$, etc.

Under functional repair with $d>0$, the per-node intra-cluster bandwidth needed from the host-cluster is lower bounded by

[TABLE]

When $d<k$, the bound gives $\gamma\geq\alpha$, i.e. the entire content of the local helper nodes must be used. For $d\geq k$, at the MBR point characterized by $\alpha=d\beta$, the bound gives $\gamma\geq(k-1)\beta$, and at the MSR point characterized by $\alpha=(d-k+1)\beta$, the bound gives $\gamma\geq 0$. The trivial bound at the MSR point is indeed optimal, since optimal (achieving equality in (1)) codes at the MSR point can be achieved by simply stacking $m$ classical $(n,k,d),(\alpha,\beta)$ MSR codes. In this case, no local help is needed for repair, and hence $\gamma=0$ is indeed optimal at the MSR point. In fact, under functional repair, the bound in (2) is optimum not just at the MSR point; we prove the converse statement that, as long as $\gamma\geq\gamma^{*}$, it is indeed possible to achieve the optimal file-size in $\eqref{eq:file_size}$ for any set of parameters, as long as there is a known upper bound on the number of repairs in the system.

We provide bounds for the parameters $\ell^{\prime}$ and $\gamma^{\prime}$ which characterize intra-cluster bandwidth from remote helper cluster under the assumption 444The values of $\alpha$ in the range $(d-k+2)\beta<\alpha\leq(d-k+1)\beta$ corresponds to the region in the trade-off between the minimum-storage operating point and the next corner point. The bound in (3) does not apply when $\alpha$ is in this range. that $\alpha\geq(d-k+2)\beta$, and $d\geq k$. Specifically we show the parameter $\ell^{\prime}$ is no less than $m$, i.e., $\ell^{\prime}=m$, and

[TABLE]

Under functional repair, RLNCs simultaneously optimize usage of both inter-cluster and intra-cluster bandwidths. Our simulations based on RLNCs indicate the tightness (achievability) of the bound in (3), under functional repair (see Fig. 3). We do not have an analytical converse for this bound.

The bounds on $\ell^{\prime}$ and $\gamma^{\prime}$ highlight the necessary trade-off between the system capacity $B^{*}$ and the remote helper intra-cluster bandwidth555We note that we quantify the remote helper intra-cluster bandwidth as $\ell^{\prime}\gamma^{\prime}$ though from Fig. 1, one gets the impression that this is $(\ell^{\prime}-1)\gamma^{\prime}$. This discrepancy arises because, in our IFG analysis (see Section III), we assume the presence of a dedicated external compute node which connects to all the $\ell^{\prime}$ helper nodes and generates the helper data. Such an assumption makes the IFG modeling symmetric with respect to the helper nodes. The choice of $\ell^{\prime}\gamma^{\prime}$ instead of $(\ell^{\prime}-1)\gamma^{\prime}$ as the amount of helper bandwidth is purely matter of convenience. The nature of results do not change even if one assumed $(\ell^{\prime}-1)\gamma^{\prime}$ as the amount of helper bandwidth. $\ell^{\prime}\gamma^{\prime}=m\gamma^{\prime}$, via parameter $\ell$, the key parameter that distinguishes our model from the classical model. Our bounds reveal the interesting fact that, while it is beneficial to increase the number of local helper nodes $\ell$ in order to improve the storage-vs-inter-cluster-repair-bandwidth trade-off, increasing $\ell$ not only increases intra-cluster repair-bandwidth in the host-cluster, but also increases the intra-cluster repair-bandwidth in the remote helper clusters. For example, if we consider MBR codes (having minimum inter-cluster-repair-bandwidth), we see that their storage-overhead approaches that of MSR codes for large $m$ as $\ell$ gets close to $m$. However, a high value of $\ell$ also increases the remote helper cluster bandwidth; indeed, $m\gamma^{\prime}$ surges as $m-\ell$ approaches $1$; see Fig. 4 for an illustration.

I-C4 Security under Passive Eavesdropping - Bounds and Codes

We study resilience of the clustered storage system against passive eavesdropping. An eavesdropper (say, Eve) gains access to the entire content of any subset of $e$ clusters, where $1\leq e\leq k$. Eve also gets to observe all the helper data downloaded for repair of nodes in these $e$ clusters. The properties of data collection and disk repair remain same as in the case of no eavesdropper. The setting is along the lines of that considered in [31], where authors study security under the classical RC framework. The maximum file size $B^{(s)}$ that can be securely stored such that Eve does not gain any information about the file is shown to be upper bounded by

[TABLE]

We also present explicit optimal secure codes for the MBR point under the setting of exact repair. Like in the case of no security, an optimal secure code is constructed by suitably precoding a combination of $m$ component codes, but this time the component codes themselves are secure codes. A code at the MBR point is constructed by combining $\ell$ secure MDS codes for the wiretap-II channel [32][33], and $(m-\ell)$ classical exact-repair secure MBR codes [31][34]. Our security results are straightforward extensions of the security results for classical RCs, given our own results for the no-security case.

I-D An Example

Consider a system consisting of $n=4$ clusters, with $m=4$ nodes/cluster, where we have the availability requirement that content from any set of $k=3$ clusters suffice for data collection. Let us consider three coding options, which permit single node repair: $(i)$ A product code consisting of two $[n=m=4,k=3]$ simple parity check codes, one across the clusters, and another within a cluster. In this case, repair happens entirely within a cluster, and there is no inter-cluster bandwidth. Note that this corresponds to the case of $d=0$ in our framework. $(ii)$ Stacking $m$ classical $(n,k,d=3)$ RCs - by this we mean that corresponding nodes from the $n$ clusters employ the classical RCs. This corresponds to the case of $\ell=0$ in our framework. $(iii)$ A $(n,k,d=3)(m=4,\ell=3)$ GRC, which is constructed as follows (see Fig. 5): We stack $m-1=3$ $[n=4,k=3]$ simple parity check codes (over the vector alphabet $\mathbb{F}_{q}^{\alpha}$) to populate coded data in all but the last nodes of all the clusters. In the last column (corresponding to the last node of all clusters), we place the sum of the three previous codes plus a classical $(n=4,k=3,d=3)(\alpha=d\beta,\beta=1)$ MBR code. Constructions of classical MBR codes appear in [35]. Data collection property of the code is straightforward. For repair of any node, the last node of each of the remote helper clusters (with the help of the remaining $3$ nodes) first extracts the MBR code, and passes the helper data for the MBR code to the replacement node. The latter regenerates the MBR code content first, and uses the helper data from the $3$ local nodes to finally recover the original stored content. The storage-overhead vs inter-cluster-repair-bandwidth trade-off achieved by these three options is shown in Fig. 6. We see that the framework of GRCs introduced here, offers operating points which are strictly better than those that can be achieved by space sharing the first two options.

We note that in the above example, the product-code solution carries the extra advantage of handling node-availability during data collection. In other words, when using the product code, one can chose to download content from any $3$ nodes from any of the $k$ clusters. This is sometimes beneficial when the $4^{th}$ node is temporarily unavailable. No such feature is present in the GRC. In the current paper, since we deal with the case of recovery from single node failure, enforcing availability (local dependence within a cluster) essentially implies restricting oneself to using product codes, and nothing else. A key insight that we wish to convey from Fig. 6. is the fact that while dealing with single node repair, it is potentially beneficial to sacrifice availability (of nodes during data collection, offered by product codes) in order to achieve operating points that are strictly better than those obtained by space sharing the two schemes.

The rest of the document is organized as follows. In Section II, we present simple proof of the file-size bound in (1) for the special cases $d=$ and $\ell=0$. As mentioned before, the general case of the file-size bound is based on the notion of information-flow graph (IFG) models; these IFG models are developed in III, followed by the derivation of the file-size bound under functional repair for general set of parameters in Section IV. The exact-repair and functional-repair code constructions appear in Section V. Bounds on parameters relating to intra-cluster bandwidth under functional repair, are discussed in Section VI. Section VII considers security under passive eaves-dropping. Finally, our conclusions and directions for future work appear in Section VIII.

II File size bound for Special Cases

In this section, we consider certain special cases of the setting of generalized regenerating codes, and identify the corresponding storage vs inter-cluster-repair-bandwidth trade-offs. The following cases are considered $1)$ $d=0$, which corresponds to the case when repair of a node is carried out entirely with the help of other local helper nodes in the cluster. $2)$ $\ell=0$, which corresponds to the case when repair of a node is carried out solely with the help of remote clusters, without taking any help from local nodes in the host-cluster.

II-A Case $d=0$ : No Inter-Cluster Help for Repair

When $d=0$, node repair is accomplished by contacting any set of $\ell$ other nodes in the host cluster. Since any set of $k$ clusters should be sufficient to decode the whole file, it follows that the file-size $B$ is upper bounded by

[TABLE]

The achievability of the above bound follows by using an $[n,k]\times[m,\ell]$ product code, with both component codes being MDS codes over $\mathbb{F}_{q}^{\alpha}$. In fact, the parameter $\alpha$ is redundant for this case; one may choose $\alpha=1$ in practice. It is clear that there is no storage vs inter-cluster-repair-bandwidth trade-off offered by this case.

II-B Case $\ell=0$: No local helper data

We next consider the special case when repair is performed without any help from nodes in the host-cluster. Let $\mathcal{C}_{m}$ denote a GRC of file-size $B$ with parameters $\{(n,k,d)(\alpha,\beta)(m,\ell=0)\}$. Let $B^{*}_{m}$ denote that maximum possible file-size for any GRC having parameters $\{(n,k,d)(\alpha,\beta)(m,\ell=0)\}$. We note that the code $\mathcal{C}_{1}$ denotes a classical RC, and under functional-repair, we know that [13]

[TABLE]

Theorem II.1

The optimal file size under the setting of functional-repair GRCs, for the case of no local helper nodes, is given by

[TABLE]

Proof:

The achievability part of the proof is straightforward; the optimal code is constructed by simply stacking $m$ classical codes $\mathcal{C}_{1}$ each of which achieves the bound in (6). By stacking, we mean that the code $\mathcal{C}_{1}$ is deployed across the corresponding nodes from all $n$ clusters. In this case, note that during node-repair, there is no pooling of content from various nodes of a remote helper cluster; repair happens as though there is only one code $\mathcal{C}_{1}$ in the system.

For showing the upper bound on the file size, we note that given a code $\mathcal{C}_{m}$ with file-size $B$ having parameters $\{(n,k,d)(\alpha,\beta)(m,\ell=0)\}$, one can construct a functional-repair classical regenerating code $\widehat{\mathcal{C}}_{1}$, also with file-size $B$, and having parameters $\{(n,k,d),(m\alpha,\widehat{\beta})\}$, where $\widehat{\beta}\leq m\beta$. For this, we simply assume the contents of all $m$ nodes of any cluster $i$ of $\mathcal{C}_{m}$, to be the contents of node $i$ of $\widehat{\mathcal{C}}_{1},1\leq i\leq n$. Clearly $\widehat{\mathcal{C}}_{1}$ retains the data collection property. For node repair in $\widehat{\mathcal{C}}_{1}$, we perform individual repairs of each of the $m$ nodes, but with the same set of $d$ remote helper clusters. In this case, we know that

[TABLE]

∎

Theorem II.1 implies that, under functional repair, the normalized trade-off between storage-overhead $nm\alpha/B$ and inter-cluster-repair-bandwidth-overhead $d\beta/\alpha$ for an $\{(n,k,d)(\alpha,\beta)(m,\ell=0)\}$ GRC is identical for any $m$; specifically, it is identical to the trade-off of an $\{(n,k,d)(\alpha,\beta)\}$ functional-repair classical RC.

III Information Flow Graph Model

In this section, we describe the information flow graph (IFG) models used to derive the various bounds in this work. The models are generalizations of the one used in [13] for the case of classical regenerating codes. Under functional repair, the problem is one of multicasting the source file to an arbitrary number of data collectors over the IFG. The IFG characterizes the data flows from the source to a data collector, and also reflects the sequence of failures and repairs in the storage system. Two models of IFGs will be used; the first one will be used in two scenarios: $1)$ to derive the trade-off between storage-overhead and inter-cluster repair-bandwidth overhead. While obtaining this trade-off, we ignore the effects of intra-cluster bandwidth, $2)$ to find the optimal local helper node intra-cluster bandwidth $\gamma$, which is needed to establish the optimal trade-off between storage-overhead and inter-cluster repair-bandwidth overhead. We wish to note that while obtaining the bound on $\gamma$, we do not impose any limitations on $\gamma^{\prime},\ell^{\prime}$, i.e, we assume that $\gamma^{\prime}=\alpha$ and $\ell^{\prime}=m$. A second related model will be used while deriving the lower bounds on the parameters $\ell^{\prime},\gamma^{\prime}$, which relate to the intra-cluster repair bandwidth needed in the remote helper clusters. In this second model, we shall assume that $\gamma=\alpha$, i.e., we ignore the effects of limited local helper-node intra-cluster bandwidth, while calculating bounds on remote helper-node intra-cluster bandwidth. We describe the two models next.

III-A IFG Model for Storage vs Inter-Cluster Repair Bandwidth Trade-off

Let $X_{i,j}$ denote the physical node $j$ in cluster $i$, $1\leq i\leq n,1\leq j\leq m$. Recall that capacity of any node is $\alpha$. In the IFG, the physical node is represented by the pair of nodes $X_{i,j}^{in}$ and $X_{i,j}^{out}$, with an edge of capacity $\alpha$ going from $X_{i,j}^{in}$ to $X_{i,j}^{out}$. The nodes $X_{i,j}^{in}$ and $X_{i,j}^{out}$ will be respectively referred to as the in-node and out-node corresponding to the physical node $X_{i,j}$. We will write $(X_{i,j}^{in}\to X_{i,j}^{out})$ to denote that there is an edge from going from $X_{i,j}^{in}$ to $X_{i,j}^{out}$. With a slight abuse of notation, we will let $X_{i,j}$ to also denote the pair $(X_{i,j}^{in},X_{i,j}^{out})$ of the graph nodes. Cluster $i$ also has an additional external node, denoted as $X_{i}^{ext}$. Each out-node $X_{i,j}^{out},1\leq j\leq m$ is connected to $X_{i}^{ext}$ via an edge of capacity $\alpha$. The external node $X_{i}^{ext}$ is used to transfer data outside the cluster, and thus serves two purposes: $1)$ it represents a single point of contact to the cluster, for a data collector which connects to this cluster, and $2)$ it represents the compute unit which generates the $\beta$ symbols for repair of any node in a different cluster.

The source node $S$ represents the original placement of the encoded source file into the $nm$ storage nodes. $S$ connects to the in-nodes of all physical storage nodes in their original state $(S\to X_{i,j}^{in}),\forall i\in[n],\forall j\in[m]$, via links of infinite capacity. The sink node $T$ represents a data collector, it connects to the external nodes of an arbitrary subset of $k$ clusters $(X_{i}^{ext}\to T)$ also via links of infinite capacity.

Each cluster at any moment has $m$ active nodes. When a physical node $X_{i,j}$ fails, it becomes inactive, and its replacement node, say $\widehat{X}_{i,j}$, becomes active instead (see Fig. 7 for an illustration). The replacement node $\widehat{X}_{i,j}$ is regenerated by downloading $\beta$ symbols from any $d$ nodes in the set $\{X_{i^{\prime}}^{ext},1\leq i^{\prime}\leq n,i^{\prime}\neq i\}$. The replacement node also connects to any subset of $\ell$ nodes in the set $\{X_{i,j^{\prime}}^{out},1\leq j^{\prime}\leq m,j^{\prime}\neq j\}$. The capacity of the links $\{(X_{i,j^{\prime}}^{out}\to\widehat{X}_{i,j}^{in}),1\leq j^{\prime}\leq m,j^{\prime}\neq j\}$ depend on whether we use the model for finding the inter-cluster-bandwidth vs storage trade-off, or we use it for finding bounds on local helper bandwidth $\gamma$. These links have capacity $\alpha$ and $\gamma$ in the former and latter cases, respectively.

In our model, recall that we focus on one repair at a time. In this scenario, along with the replacement of $X_{i,j}$ with $\widehat{X}_{i,j}$, we will also copy all the remaining $m-1$ nodes, as they are, in the cluster $i$, and represent them with new identical pair of nodes $(X_{i,j^{\prime}}^{in},X_{i,j^{\prime}}^{out}),1\leq j^{\prime}\leq m,j^{\prime}\neq j$. We shall also a have a new external node for the cluster, which connects to the new $m$ out-nodes. Thus, in the IFG modeling, we say that the entire old cluster (where the failed node resides) becomes inactive, and gets replaced by a new active cluster. For either data collection or repair, we connect to external nodes of the active clusters. Note that, at any point in time, a physical cluster contains only one active cluster in the IFG, and $f_{i}$ inactive clusters in the IFG, where $f_{i}\geq 0$ denotes the total number of failures and repairs experienced by the various nodes in the cluster. We shall use the notation $\mathcal{X}_{i}(t),0\leq t\leq f_{i}$ to denote the cluster that appears in IFG after the $t^{\text{th}}$ repair associated with cluster $i$. The clusters $\mathcal{X}_{i}(0),\ldots,\mathcal{X}_{i}(f_{i}-1)$ are inactive, while $\mathcal{X}_{i}(f_{i})$ is active, after $f_{i}$ repairs. The nodes of $\mathcal{X}_{i}(t)$ will be denoted by $X_{i,j}^{in}(t),X_{i,j}^{out}(t),{X}_{i}^{ext}(t),1\leq j\leq m$. With a slight abuse of notation, we will let $\mathcal{X}_{i}(t)$ to also denote the collection of all $2m+1$ nodes in this cluster. We write $X_{i,j}(t)$ to denote the pair $(X_{i,j}^{in}(t),X_{i,j}^{out}(t))$; again, with a slight abuse of notation, we shall use $X_{i,j}(t)$ to also denote the node $j$ in cluster $i$ after the $t^{\text{th}}$ repair (in cluster $i$). We further use notation $Fam(i)$ to denote the union of all nodes in all inactive clusters, and the active cluster, corresponding to the physical cluster $i$ after $t$ repairs in cluster $i$, i.e., $Fam(i)=\cup_{t=0}^{f_{i}}\mathcal{X}_{i}(t)$. We have avoided indexing $Fam(i)$ with the parameter $t$ as well, to keep the notation simple. The value of $t$ in our usage of the notation $Fam(i)$ will be clear from the context.

III-B IFG Model for Finding Bounds on Intra-Cluster Repair Bandwidth

We now describe the model used to obtain lower bounds on the parameters $\gamma^{\prime},\ell^{\prime}$. Unlike in the case of bounds for file-size $B$ and local helper node intra-cluster bandwidth $\gamma$, where we also show converses, for $\gamma^{\prime}$ and $\ell^{\prime}$ we do not provide converses to the lower bounds. When only dealing with lower bounds, we can significantly simplify the model described above. In addition to making these simplifications, we also add some structure to the model to enable usage of $\gamma^{\prime}$ and $\ell^{\prime}$. We describe these changes next.

In the second model, each physical node $X_{i,j}$ is again represented by the pair of nodes $(X_{i,j}^{in},X_{i,j}^{out})$, such that the edge $(X_{i,j}^{in}\to X_{i,j}^{out})$ has capacity $\alpha$. In this model, an external node(s) is(are) added dynamically to a cluster whenever it aids in either data collection or repair of another node. Each instance of an external node is used exactly once, either for a repair or a data collection operation. Whenever a physical node $X_{i,j}$ fails, we say that it becomes inactive, and its replacement node, say $\widehat{X}_{i,j}=(\widehat{X}_{i,j}^{in},\widehat{X}_{i,j}^{out})$, becomes active in the same cluster. The remaining $m-1$ nodes are not replicated, as in the previous model. Thus, in the second model, there is only a single graph cluster corresponding to a physical cluster. On the graph representation, the replacement node is visually linked to the replaced failed node with a red dotted line (see Fig. 8 for an example).

We explain the repair and data collection operation in the IFG in more detail next. During repair, the replacement node $\widehat{X}_{i,j}$ connects to any subset of $\ell$ active nodes in the same cluster via links of capacity $\alpha$. It also downloads $\beta$ symbols each from $d$ remote helper clusters via their external nodes. If replacement node $\widehat{X}_{i,j}$ downloads helper data from cluster $i^{\prime}$, an external node, $X_{i^{\prime},X_{i,j}}^{ext}$, is added to the IFG, such that the edge $(X_{i^{\prime},X_{i,j}}^{ext}\to\widehat{X}_{i,j}^{in})$ has capacity $\beta$. External node $X_{i^{\prime},X_{i,j}}^{ext}$ also connects locally to a subset of $\ell^{\prime}$ active out-nodes of cluster $i^{\prime}$ via links of capacity $\gamma^{\prime}$. Note how we index the external node of cluster $i^{\prime}$ that aids in the repair of ${X}_{i,j}$. Every time cluster $i^{\prime}$ acts as a remote helper cluster toward the repair of any node, we add a new external node in a manner similar to $X_{i^{\prime},X_{i,j}}^{ext}$. Finally, a data collector, $T$, connects to cluster $i$ via the external node $X_{i,T}^{ext}$, which in turn connects to all $m$ active out-nodes in the cluster via links of capacity $\alpha$. We index the external node also with $T$ since the $m$ active nodes that form part of the cluster evolves over time.

In comparison with previous model, we do not time-index the sequence of failures in the current model. This is because, in our proof of bounds for $\gamma^{\prime}$ and $\ell^{\prime}$, we only consider system evolutions in which each node fails at most once. In this case, we find it convenient simply to denote the replacement node of $X_{i,j}$ as $\widehat{X}_{i,j}$.

IV File size bound for General Parameters

In this section, we derive the file-size bound in (1) under the setting of functional repair, for arbitrary set of code parameters. We further use this bound to characterize the storage-overhead vs inter-cluster-repair-bandwidth-overhead trade-off. Intra-cluster bandwidth is ignored in this section. Thus, for the repair of any node, the entire content of $\ell$ local helper nodes can be used; similarly, the entire content ($m\alpha$ symbols) of each remote helper cluster is used to generate its $\beta$ helper symbols.

Theorem IV.1

The file size $B$ of a functional repair generalized regenerating code having parameters $\{(n,k,d)$ $(\alpha,\beta)$ $(m,\ell)\}$ is upper bounded by

[TABLE]

Further, if there is a known upper bound on the number of repairs that occur for the duration of operation of the system, the above bound is sharp, i.e., $B^{*}$ gives the functional repair storage capacity of the system.

Proof:

The proof technique is similar to the proof of the bound under functional repair for the setting of classical regenerating codes [13]. The problem of functional repair is one of multicasting, and thus for finding the desired upper bound on the file-size, it is enough if we exhibit a cut in an IFG, for a specific sequence of failures and repairs, which separates the source from the sink, such that the value of the cut is the desired upper bound. We shall then show that, for any valid IFG, independent of the specific sequence of failures and repairs, $B^{*}$ is indeed a lower bound on the minimum possible value of any $S-T$ cut. The achievability result, when there is known upper bound on the number of failures and repairs, will then follow from results in network coding  [28].

We begin with the proof of the upper bound. We consider a sequence of $k(m-\ell)$ failures and repairs, as follows: Physical nodes $X_{i,\ell+1},X_{i,\ell+2},\ldots X_{i,m}$ fail in this order in cluster $i=1$, then in cluster $i=2$, and so on, until cluster $i=k$. In the IFG, (see Section III-A), this corresponds to the sequence of failures of nodes $X_{1,\ell+1}(0),X_{1,\ell+2}(1),\ldots,X_{1,m}(m-\ell-1),X_{2,\ell+1}(0),\ldots,X_{2,m}(m-\ell-1),\ldots,X_{k,m}(m-\ell-1)$, in the respective order. The replacement node $X_{i,\ell+t}(t)$ for $X_{i,\ell+t}(t-1),1\leq t\leq m-\ell$ draws local helper data from $X_{i,1}(t-1),X_{i,2}(t-1),\cdots,X_{i,\ell}(t-1)$, and remote helper data from the clusters $\mathcal{X}_{1}(m-\ell),\cdots,\mathcal{X}_{i-1}(m-\ell)$ and from some set of $d-\min\{i-1,d\}=(d-i+1)^{+}$ other active clusters in the IFG. An example is shown in Fig. 9 for a set of system parameters that is same as those used in Fig. 7.

Let data collector $T$ connect to clusters $\mathcal{X}_{1}(m-\ell),\ldots,\mathcal{X}_{k}(m-\ell)$. Consider the $S-T$ cut consisting of the following edges of the IFG:

• •

$\{(X_{i,j}^{in}(0)\to X_{i,j}^{out}(0),i\in[k],j\in[\ell]\}$. Total capacity of these edges is $kl\alpha$.

• •

For each $i\in[k],t\in[m-\ell]$, either the set of edges $\{(X_{i^{\prime}}^{ext}(0)\to X_{i,\ell+t}^{in}(t)),i^{\prime}\in\{$remote helper cluster indices for the replacement node $X_{i,\ell+t}^{in}(t)\}\backslash[\min\{i-1,d\}]\}$, or the edge $(X_{i,\ell+t}^{in}(t)\to X_{i,\ell+t}^{out}(t))$. Between the two possibilities, we pick the one which has smaller capacity. In this case, the total capacity of this part of the cut is given by $\sum_{i=1}^{k}\sum_{j=\ell+1}^{m}\min\{\alpha,(d-\min\{i-1,d\})\beta\}=(m-\ell)\sum_{i=1}^{k}\min\{\alpha,(d-i+1)^{+}\beta\}$.

The value of the cut is given by $kl\alpha+(m-\ell)\sum_{i=1}^{k}\min\{\alpha,(d-i+1)^{+}\beta\}=B^{*}$, which proves our upper bound. In the example on Fig. 9 for $(n=3,k=2,d=2)(m=2,\ell=1)$, first, $m-\ell=1$ node fails in cluster $1$ and downloads helper data from clusters $2,3$, second, a node fails in cluster $2$ and downloads helper data from clusters $1,3$. The data collector connects to clusters $1,2$. A minimal cut for $2\beta\leq\alpha$ is shown on the figure, and has value $2\alpha+3\beta=B^{*}$.

We next show that for any valid IFG (independent of the specific sequence of failures and repairs), $B^{*}$ is indeed a lower bound on the minimum possible value of any $S-T$ cut. Consider any $S-T$ cut, and let IFGS and IFGT denote the two resultant disconnected parts of the IFG corresponding to the nodes $S$ and $T$, respectively. Since node $T$ connects to $k$ external nodes via links of infinite capacity, we only consider cuts such that IFGT has at least $k$ external nodes corresponding to active clusters. Next, we observe that the IFG is a directed acyclic graph, and hence there exists a topological sorting of nodes of the graph such that an edge exists between two nodes $A$ and $B$ of the IFG only if $A$ appears before $B$ in the sorting. Further, we consider a topological sorting such that all in-, out- and external nodes of the cluster $\mathcal{X}_{i}(\tau)$ appear together in the sorted order, $\forall i,\tau$.

Now, consider the sequence $\mathcal{E}$ of all the external nodes (which are part of both active and inactive clusters) in IFGT in their sorted order. Let $Y_{1}$ denote the first node in this sequence. Without loss of generality let $Y_{1}\in Fam(1)$. Next, consider the subsequence of $\mathcal{E}$ which is obtained after excluding all the external nodes in $Fam(1)$ from $\mathcal{E}$. Let $Y_{2}$ denote the first external node in this subsequence. We continue in this manner until we find the first $k$ external nodes $\{Y_{1},Y_{2},\ldots,Y_{k}\}$ in $\mathcal{E}$, such that each of the $k$ nodes corresponds to a distinct physical cluster. Once again, without loss of generality, we assume that $Y_{i}\in Fam(i),2\leq i\leq k$. Let us assume that $Y_{i}=X_{i}^{ext}(t_{i})$, for some $t_{i}$. Now, consider the $m$ out-nodes $X_{i,1}^{out}(t_{i}),\ldots,X_{i,m}^{out}(t_{i})$ that connect to $X_{i}^{ext}(t_{i})$. Among these $m$ out-nodes, let $a_{i},0\leq a_{i}\leq m$ denote the number of out-nodes that appear in IFGS. Without loss of generality let these be the nodes $X_{i,1}^{out}(t_{i}),X_{i,2}^{out}(t_{i}),\ldots,X_{i,a_{i}}^{out}(t_{i})$. Next, corresponding to the out-node $X_{i,j}^{out}(t_{i}),a_{i}+1\leq j\leq m$, consider its past versions $\{X_{i,j}^{out}(t),t<t_{i}\}$ in the IFG, and let $X_{i,j}^{out}(t_{i,j})$, for some $t_{i,j}\leq t_{i}$ denote the first sorted node that appears666It may be noted that even though $X_{i,j}^{out}(t_{i,j})$ appears in IFGT, the corresponding external node $X_{i}^{ext}(t_{i,j})$ appears in IFGS. This is due to our assumption that $Y_{i}=X_{i}^{ext}(t_{i})$ is the first external node, corresponding to physical cluster $i$, that appears in IFGT. in IFGT. Without loss of generality, let us also assume that the nodes $\{X_{i,j}^{out}(t_{i,j}),a_{i}+1\leq j\leq m\}$ are sorted in the order $X_{i,a_{i}+1}^{out}(t_{i,a_{i}+1}),X_{i,a_{i}+2}^{out}(t_{i,a_{i}+2}),\ldots,X_{i,m}^{out}(t_{i,m})$. An illustration is provided in Fig. 10.

To obtain a lower bound on the value of the $S-T$ cut, we make the following observations:

• •

The $a_{i}$ edges $\{(X_{i,j}^{out}(t_{i})\to X_{i}^{ext}(t_{i})),1\leq j\leq a_{i}\}$ are part of the cut. These contribute a total value of $a_{i}\alpha$.

• •

For any node $X_{i,j}^{out}(t_{i,j}),a_{i}+1\leq j\leq m$, if the corresponding in-node $X_{i,j}^{in}(t_{i,j})$ belongs to IFGS, then the edge $(X_{i,j}^{in}(t_{i,j})\to X_{i,j}^{out}(t_{i,j}))$ appears in the cut, and contributes a value of $\alpha$ to the cut. Now, consider the case when the in-node $X_{i,j}^{in}(t_{i,j})$ belongs to IFGT. In this case, consider the following two sub cases:

• –

The node $X_{i,j}(t_{i,j})$ is not a replacement node: This means that, either the edge $(X_{i,j}^{out}(t_{i,j}-1)\to X_{i,j}^{in}(t_{i,j}))$ appears in the cut, if $t_{i,j}>0$, or the edge $(S\to X_{i,j}^{in}(t_{i,j}))$ appears in the cut, if $t_{i,j}=0$. In any case, the contribution to the overall value of the cut is at least $\alpha$.

• –

The node $X_{i,j}(t_{i,j})$ is a replacement node of $X_{i,j}(t_{i,j}-1)$: We know that $\ell$ local helper nodes and $d$ external nodes are involved in repair. It is straightforward to see that out of the $\ell$ local helper nodes, at most $(j-1)$ belong to IFGT. To see this, note that the potential candidates for the local helper nodes that appear in IFGT correspond to the physical nodes777It may be noted that we count the physical nodes $X_{i,1},\ldots,X_{i,a_{i}}$ among the possible set of local helpers, although we assume that $X_{i,j}(t_{i}),1\leq j\leq a_{i}$ appears in IFGS. This is because, we cannot discount the possibility that $X_{i,j}(t_{i,j^{\prime}}-1)$ appears in IFGT, for $j\leq a_{i},j^{\prime}>a_{i}$. $X_{i,1},X_{i,2},\ldots,X_{i,j-1}$. The version of the physical node $X_{i,j^{\prime}},j^{\prime}>j$ , if it aids in the repair process, appears in IFGS because of our definition of $X_{i,j^{\prime}}(t_{i,j^{\prime}})$. Next, note that out of the $d$ external nodes, at most $(i-1)$ belong to IFGT. In this case, the contribution to the value of the cut, due to the edges that aid in repair, is lower bounded by $(\ell-j+1)^{+}\alpha+(d-(i-1))^{+}\beta$.

Based on the observations above, the value of the cut is lower bounded by

[TABLE]

for any $a_{i},0\leq a_{i}\leq m$. This completes the proof of the converse. ∎

Definition 1** (Optimal Code)**

Code $\mathcal{C}_{m}$ is said to be optimal or capacity achieving, if its file-size $B=B^{*}$, where $B^{*}$ is as given in Theorem IV.1.

IV-A Minimum Storage and Minimum Inter-Cluster Bandwidth Operating Points

We now define the MSR and MBR operating points based on the bound in Theorem IV.1, whenever $d>0$. The operating point $d\beta=\alpha$ will be identified with the MBR operating point. An optimal code $\mathcal{C}_{m}$ at the MBR operating point will be referred to as an MBR code. To see the rationale behind the definition of the MBR operating point, we recall our assumption that whenever $d>0$, the encoding function does not introduce any dependency among the content of the nodes of a cluster. In this case, assuming that $B$ symbols of the uncoded file are uniformly distributed over $\mathbb{F}_{q}^{B}$, it is straightforward to see the necessity of the condition $d\beta\geq\alpha$ for any code $\mathcal{C}_{m}$.

Towards defining the MSR point, we note that $B\leq\ell k\alpha+(m-\ell)\min(d,k)\alpha$. The MSR code is an optimal code having $B=\ell k\alpha+(m-\ell)\min(d,k)\alpha$, and has the lowest possible inter-cluster repair bandwidth. It is clear that the parametrization of the MSR operating point depends on relation between $d$ and $k$. For $d\geq k$, the optimal file-size of $B=mk\alpha$ is achievable only if $(d-k+1)\beta\geq\alpha$, and thus the operating point $(d-k+1)\beta=\alpha$ will be identified with the MSR operating point. For $1\leq d\leq k-1$, note that $B\leq\ell k\alpha+(m-\ell)d\alpha$, and equality is achieved only if $\beta\geq\alpha$. The operating point $\alpha=\beta$ will be identified with the MSR operating point for the case when $1\leq d\leq k-1$.

Based on the definition of the MSR and MBR operating points, we note that for $d\geq k$, the range of $\alpha$ considered while plotting the trade-off is given by $(d-k+1)\beta\leq\alpha\leq d\beta$, and when $1\leq d\leq k-1$, the range of $\alpha$ is given by $\beta\leq\alpha\leq d\beta$.

V Code Constructions

In this section, we describe our optimal code constructions. Two constructions are presented; the first one is an instance of an exact repair code, and results in optimal codes at the MSR and MBR points under the setting of generalized regenerating codes; the second construction is a functional-repair regenerating code. Both codes can withstand any number of repairs for the duration of operation of the system. The exact repair code withstands any number of repairs by definition, since after each repair the data on all nodes is the same as at the start of system operation. This logic does not hold for functional repair codes, because the repaired node content is generally different from the original one. Network-coding based achievability proofs for functional-repair work only if there is a known upper bound on the number of repairs that occur over the lifetime of the system. Our functional repair code relies on the construction in [30], which allows our code to operate for arbitrarily many repairs. For both constructions, we rely on existing optimal classical regenerating codes that are linear. By a linear regenerating code, we mean that both encoding and repair are performed via linear combinations of either the input or the coded symbols, respectively. The first construction generates an optimal $(n,k,d)(\alpha,\beta)(m,\ell)$ code for any $m,\ell\leq m-1,1\leq d\leq n-1$, whenever an optimal $(n,k,\min(k,d))(\alpha,\beta)$ classical exact repair linear regenerating code exists. Our functional repair code construction is limited to the case $\ell=m-1,d\geq k$.

For a linear $(n,k,d)(\alpha,\beta)$ classical regenerating code that encodes a data file of size $B$ symbols, one can associate a generator matrix $G$ of size $B\times n\alpha$. Without loss of generality, the first $\alpha$ columns of $G$ generates the content of node $1$, and so on. We say that two $(n,k,d)(\alpha,\beta)$ classical linear regenerating codes $\mathcal{C}_{1}$ and $\mathcal{C}_{2}$, having generator matrices $G_{1}$ and $G_{2}$ are identical, if $G_{1}=G_{2}$. Since we assume that repair is also a linear operation, it follows that the set of linear functions that define repair operations can be taken the same for $\mathcal{C}_{1}$ and $\mathcal{C}_{2}$ that are identical. The following lemma will be used while we prove optimality of the exact-repair construction. The proof of the lemma is straightforward, and is omitted.

Lemma V.1

Let $\{\mathcal{C}_{i},1\leq i\leq s,s\geq 1\}$ denote identical $(n,k,d)(\alpha,\beta)$ classical exact repair linear regenerating codes. Let ${\bf c}_{i}\in\mathbb{F}_{q}^{n\alpha}$ denote a generic codeword of $\mathcal{C}_{i},1\leq i\leq s$, where the first $\alpha$ symbols of ${\bf c}_{i}$ are the content of node $1$, and so on. Suppose that we are given $a_{i}\in\mathbb{F}_{q},1\leq i\leq s$ such that not every $a_{i}$ is [math], define a new $n$-length array code $\mathcal{C}$ over $\mathbb{F}_{q}^{\alpha}$ as $\mathcal{C}=\{\sum_{i=1}^{s}a_{i}{\bf c}_{i},{\bf c}_{i}\in\mathcal{C}_{i}\}$. Then, the code $\mathcal{C}$ is an $(n,k,d)(\alpha,\beta)$ classical exact repair linear regenerating code over $\mathbb{F}_{q}$, and is identical to $\mathcal{C}_{i},1\leq i\leq s$.

V-A Exact Repair Code Construction

We begin with a description of the code, and then show its data collection and repair properties. The construction itself is a generalization of the example presented in Section I-D.

Construction V.2

Let $\mathcal{C}_{j},1\leq j\leq\ell$ denote $[n,k]$ MDS array codes over $\mathbb{F}_{q}^{\alpha}$. The amount of data that can be encoded with these $\ell$ codes is $\ell k\alpha$. Next, let $\mathcal{C}_{j},\ell+1\leq j\leq m$ denote $(n,k,d^{\prime}=\min(d,k))(\alpha,\beta)$ classical exact repair linear regenerating codes (also over $\mathbb{F}_{q}$), each having a file size $B^{\prime}=\sum_{i=0}^{k-1}\min(\alpha,(d^{\prime}-i)\beta)$. We require that all the codes $\mathcal{C}_{j},\ell+1\leq j\leq m$ are identical. For encoding, we first divide the data file of size $B^{*}=\ell k\alpha+(m-\ell)B^{\prime}$ into $m$ stripes, such that first $\ell$ have size $k\alpha$, and the last $m-\ell$ have size $B^{\prime}$. Stripe $j,1\leq j\leq m$ is encoded by $\mathcal{C}_{j}$ to generate the coded symbols ${\bf c}_{j}=[c_{1,j},c_{2,j},\ldots,c_{n\alpha,j}]^{T}$. Next, consider an $m\times m$ invertible matrix $A$ over $\mathbb{F}_{q}$ such that the first the $\ell$ rows of $A$ generate an $[m,\ell]$ MDS code $\mathbb{F}_{q}$. Let matrix $A$ be decomposed as

[TABLE]

Thus, the matrix $E_{\ell\times m}$ generates an $[m,\ell]$ MDS code. The coded data stored in the various clusters is generated as follows:

[TABLE]

The content of node $j$ in cluster $i$ is given by $[c^{\prime}_{(i-1)\alpha+1,j},c^{\prime}_{(i-1)\alpha+2,j},\ldots,c^{\prime}_{i\alpha,j}]^{T},1\leq i\leq n,1\leq j\leq m$. This completes the description of the construction. A pictorial overview of the description appears in Fig. 11.

We next prove optimality property of Construction V.2.

Theorem V.3

The code described in Construction V.2 is an optimal exact repair generalized regenerating code, for any $m,\ell\leq m$. The optimal code can be constructed whenever an optimal $(n,k,d^{\prime}=\min(d,k))(\alpha,\beta)$ exact repair linear regenerating code exists, having a file size $B^{\prime}=\sum_{i=0}^{k-1}\min(\alpha,(d^{\prime}-i)\beta)$.

Proof:

It is clear that the code in Construction V.2 has a file size $B^{*}$, where $B^{*}$ is as given in Theorem IV.1. Further, the data collection property of the code is also straightforward to check, and this essentially follows from the facts that $1)$ the matrix $A$ is invertible, and $2)$ each of the codes $\mathcal{C}_{i},1\leq i\leq m$ is uniquely decodable given its coded data belonging to any $k$ clusters. Towards examining the repair property of the code let us rewrite (20) as follows:

[TABLE]

where ${\bf C}_{MDS}=[{\bf c}_{1}\ \cdots\ {\bf c}_{\ell}]$ and ${\bf C}_{regen}=[{\bf c}_{\ell+1}\ \cdots\ {\bf c}_{m}]$. The matrices ${\bf c}^{(i)}_{MDS}$ and ${\bf c}^{(i)}_{regen},1\leq i\leq n$ denote rows $(i-1)\alpha+1,\ldots,i\alpha$ of ${\bf C}_{MDS}$ and ${\bf C}_{regen}$, respectively. Let us also expand the decomposition of matrix $A$ in (19) further as follows:

[TABLE]

where ${\bf e}_{j}^{T}$ and ${\bf f}_{j}^{T},1\leq j\leq m$ denote the $j^{\text{th}}$ column of the matrices $E$ and $F$, respectively. Based on (27) and (35), it can be seen that the content of node $j$ in cluster $i$ is given by

[TABLE]

Given the notation above, without loss of generality, consider repairing node $\ell+1$ in cluster $1$ with the help of $1)$ the first $\ell$ local nodes in cluster $1$ and $2)$ clusters $2,\ldots,d^{\prime}+1$. Let us first examine the role of the $\ell$ local nodes in the repair process. Let $E^{\prime}$ and $F^{\prime}$ denote the first $\ell$ columns of $E$ and $F$, respectively. By assumption, $E$ generates an $[m,\ell]$ MDS code, and hence the submatrix $E^{\prime}$ is invertible. In this case, the content from the $\ell$ local nodes can be put together to generate

[TABLE]

where ${\bf\widehat{f}}_{j}^{T}=F^{\prime}E^{\prime-1}{\bf e}_{\ell+1}^{T}$. Thus, the local helper nodes serve to recover the part corresponding to the MDS-codes’ components given by ${\bf c}^{(1)}_{MDS}{\bf e}_{\ell+1}^{T}$. However, the regenerating-codes’ components ${\bf c}^{(1)}_{regen}{\bf\widehat{f}}_{\ell+1}^{T}$ differs from the original ${\bf c}^{(1)}_{regen}{\bf{f}}_{\ell+1}^{T}$.

Let us next examine the role of the $d^{\prime}$ remote helper clusters. We know that the data stored in cluster $i$ is given by $\left[{\bf c}^{(i)}_{MDS}\ \ {\bf c}^{(i)}_{regen}\right]A$. Since matrix $A$ is invertible, the vector ${\bf c}^{(i)}_{regen}$ can be recovered from this. Now, if we define the code $\widehat{\mathcal{C}}_{i}=\left\{\sum_{j=1}^{m-\ell}(f_{j}-\widehat{f}_{j}){\bf c}_{\ell+j},{\bf c}_{\ell+j}\in\mathcal{C}_{\ell+j}\right\}$, we know from Lemma V.1 that $\widehat{\mathcal{C}}_{i}$ is an $(n,k,d^{\prime})(\alpha,\beta)$ classical exact repair linear regenerating code, which is identical to $\mathcal{C}_{j},\ell+1\leq j\leq m$. Thus, cluster $i,2\leq i\leq d^{\prime}+1$ generates and passes the helper data ($\beta$ symbols) toward the repair of the first vector symbol for the code $\widehat{\mathcal{C}}_{i}$. The replacement node regenerates ${\bf c}^{(1)}_{regen}({\bf f}_{j}^{T}-\widehat{{\bf f}}_{j}^{T})$ using the helper data from the $d^{\prime}$ remote clusters, and combines it with the local helper data (see (46)) to correct the regenerating-codes’ components, and restore the content of the lost node. A pictorial illustration of the repair process is shown in Fig. 12. ∎

V-B A Functional Repair Code for Arbitrary Number of Failures

In this section, we show the existence of optimal functional repair codes over a finite field that can tolerate an arbitrary number of repairs for the duration of operation of the system. We show the existence for any $(n,k,d)(\alpha,\beta)(m,\ell=m-1)$. The code construction is similar to the one used in the example in Section I-D, and combines $m-1$ MDS array codes $\mathcal{C}_{1},\ldots,\mathcal{C}_{m-1}$ with an $(n,k,d)(\alpha,\beta)$ functional repair code $\mathcal{C}_{m}$ for the classical setting. The code $\mathcal{C}_{m}$ is one that can tolerate an arbitrary number of repairs for the duration of operation of the system. The following lemma is a direct consequence of Theorem $3$ and the description in Section $V$ of [30], and guarantees the existence of the code $\mathcal{C}_{m}$ that we use here.

Lemma V.4

For any $(n,k,d)(\alpha,\beta)$, there exists an optimal deterministic classical functional repair linear regenerating code over $\mathbb{F}_{q}$ that can tolerate an arbitrary number of repairs for the duration of operation of the system, whenever $q>q_{0}$, where $q_{0}$ is entirely determined by the parameters $(n,k,d)(\alpha,\beta)$, and is independent of the number of repairs performed over the lifetime of the code.

In the above lemma, by a deterministic regenerating code, we mean that the regenerated data corresponding to a repair operation of a given physical node is uniquely determined given the content of the helper nodes. As we shall see, the fact the code is deterministic is important to ensure the data collection property of our functional repair construction.

Below, we first describe the code construction, along with the repair procedure, and then show the optimality property of the code. In the following construction, we assume that the characteristic of the finite field $\mathbb{F}_{q}$ is $2$; i.e., $q=2^{w}$ for some $w>0$. The assumption is made only for the ease of description, the construction can be modified to accommodate finite fields of any characteristic.

Construction V.5

Let $\mathcal{C}_{j},1\leq j\leq m-1$ denote $[n,k]$ MDS array codes over $\mathbb{F}_{q}^{\alpha}$. The amount of data that can be encoded with these $m$ codes is $(m-1)k\alpha$. Next, let $\mathcal{C}_{m}$ denote an $(n,k,d^{\prime}=\min(d,k))(\alpha,\beta)$ classical functional repair linear regenerating code whose existence is guaranteed by Lemma V.4. The code $\mathcal{C}_{m}$ has a file size $B^{\prime}=\sum_{i=0}^{k-1}\min(\alpha,(d^{\prime}-i)\beta)$. For encoding, we first divide the data file of size $B^{*}=\ell k\alpha+(m-\ell)B^{\prime}$ into $m$ stripes, such that first $m-1$ have size $k\alpha$, and the last one has size $B^{\prime}$. Stripe $j,1\leq j\leq m$ is encoded by $\mathcal{C}_{j}$ to generate the coded symbols ${\bf c}_{j}=[c_{1,j},c_{2,j},\ldots,c_{n\alpha,j}]^{T}$. The arrangement of coded data in the various nodes is identical to that in the example in Section I-D. Thus, node $j,1\leq j\leq m-1$ in cluster $i$ stores the vector $[c_{(i-1)\alpha+1,j},c_{(i-1)\alpha+2,j},\ldots,c_{i\alpha,j}]^{T}$. Node $m$ in cluster $i$ stores the sum of the content of $m-1$ nodes with symbols $[c_{(i-1)\alpha+1,m},c_{(i-1)\alpha+2,m},\ldots,c_{i\alpha,m}]^{T}$ of the regenerating code $\mathcal{C}_{m}$, i.e., content of node $m$ is given by $[\sum_{j=1}^{m}c_{(i-1)\alpha+1,j},\sum_{j=1}^{m}c_{(i-1)\alpha+2,j},\ldots,\sum_{j=1}^{m}c_{i\alpha,j}]^{T}$. This completes the description of the initial layout of the coded data. Since the code is a functional repair code, the code description is not complete unless we specify the procedure for node repair, as well. We do this next.

Node Repair:* Let $Y_{i,j}(t)\in\mathbb{F}_{q}^{\alpha}$ denote the content of node $j$ in cluster $i$, after the $t^{\text{th}},t\geq 0$ repair, anywhere in the system. The quantities $\{Y_{i,j}(0),1\leq i\leq n,1\leq j\leq m\}$ denote the initial content present in the system, and are as described above. The repair procedure is such that the vector $[\sum_{j=1}^{m}Y_{1,j}(t),\sum_{j=1}^{m}Y_{2,j}(t),\ldots,\sum_{j=1}^{m}Y_{n,j}(t)]^{T}$ remains as a valid codeword of the functional repair regenerating code $\mathcal{C}_{m}$, for every $t\geq 0$ (to be proved in Theorem V.6). Clearly, the above statement is true for $t=0$. The repair procedure can be described recursively as follows: Let the $t^{\text{th}}$ repair be associated with node $i^{\prime}$ in cluster $j^{\prime}$. Each of the $d^{\prime}$ remote helper clusters, say $i$, internally computes $\sum_{j=1}^{m}Y_{i,j}(t-1)$, and passes the $\beta$ symbols toward the repair of $\sum_{j=1}^{m}Y_{i^{\prime},j}(t-1)$. The replacement node first of all regenerates $\widehat{Y}_{i^{\prime}}(t-1)$, as the replacement of $\sum_{j=1}^{m}Y_{i^{\prime},j}(t-1)$, given the helper data from the $d$ remote clusters. Next, since $\ell=m-1$, the replacement node gets access to local helper data $\{Y_{i^{\prime},j}(t-1),1\leq j\leq m,j\neq j^{\prime}\}$. The content that is eventually stored in the replacement node is computed as follows:*

[TABLE]

Also, for any $(i,j)\neq(i^{\prime},j^{\prime})$, we assume that

[TABLE]

This completes the description of the repair process and the code construction.

In the following theorem, we argue the optimality property of the above construction. Specifically, we show that the code retains the functional repair and data collection properties, after every repair. We assume that the data collector is aware of the entire repair-history of the system. By this we mean that the data collector is aware of $1)$ the exact sequence of $t$ failures and repairs that has happened in the system, and $2)$ the indices of the remote helper clusters that aided in each of the $t$ repairs.

Theorem V.6

The code described in Construction V.5 is an optimal $(n,k,d)(\alpha,\beta)(m,\ell=m-1)$ functional repair generalized regenerating code.

Proof:

It is clear that the code in Construction V.5 has a file-size $B^{*}$, as given by Theorem IV.1. Toward showing that the code retains functional repair property, it is sufficient if we show that the vector $[\sum_{j=1}^{m}Y_{1,j}(t),$ $\sum_{j=1}^{m}Y_{2,j}(t),\ldots,\sum_{j=1}^{m}Y_{n,j}(t)]^{T}$ remains as a valid codeword of the functional repair regenerating code $\mathcal{C}_{m}$, for every $t\geq 0$. We do this inductively. Clearly, the statement is true for $t=0$. Let us next assume that the statement is true for $t=t^{\prime}\geq 0$, and show its validity for $t=t^{\prime}+1$. Assume that the $(t^{\prime}+1)^{\text{th}}$ repair is associated with node $j^{\prime}$ in cluster $i^{\prime}$. The relation between the content of the various nodes before and after the $(t^{\prime}+1)^{\text{th}}$ repair are obtained via (47) and (48). In this case, the quantities $\{\sum_{j=1}^{m}Y_{i,j}(t^{\prime}+1),1\leq i\leq n\}$ are given by

[TABLE]

where $(a)$ follows from our assumption that the finite field $\mathbb{F}_{q}$ has characteristic $2$. Now, recall that $\widehat{Y}_{i^{\prime}}(t^{\prime})$ is the replacement of $\sum_{j=1}^{m}Y_{i^{\prime},j}(t^{\prime})$, which is regenerated using the helper data generated using $d$ elements of the set $\{\sum_{j=1}^{m}Y_{i,j}(t^{\prime}),1\leq i\leq n,i\neq i^{\prime}\}$. Combining with the induction hypothesis for $t=t^{\prime}$, it follows that the induction statement holds good for $t=t^{\prime}+1$ as well. This completes the proof of functional repair property of the code.

Let us next see how data collection is accomplished after $t,t\geq 0$ repairs in the system. Without loss of generality assume that a data collector connects to clusters $1,2,\ldots,k$, and accesses $\{Y_{i,j}(t),1\leq i\leq k,1\leq j\leq m\}$. The data collector as a first step computes the vector $[\sum_{j=1}^{m}Y_{1,j}(t),\sum_{j=1}^{m}Y_{2,j}(t),\ldots,\sum_{j=1}^{m}Y_{k,j}(t)]^{T}$, and uses this to decode the data corresponding to the code $\mathcal{C}_{m}$. Now, recall the fact that the code $\mathcal{C}_{m}$ is deterministic, and also our assumption that the data collector is aware of the entire repair-history of the system. In this case, having decoded $\mathcal{C}_{m}$, using (47) and (48), the data collector can iteratively recover $\{Y_{i,j}(t^{\prime}),1\leq k\leq,1\leq j\leq m\}$, for $t\geq t^{\prime}\geq 0$ by starting at $t^{\prime}=t$ and proceeding backwards until the content at $t^{\prime}=0$ is recovered (essentially, we are rewinding the system by eliminating the effects of all the repairs, starting from the last one and proceeding backwards in time). Finally, from Construction V.5, we know that the content $\{Y_{i,j}(0),1\leq k\leq,1\leq j\leq m-1\}$ is the stacked coded data corresponding to the $m-1$ $[n,k]$ MDS coded $\mathcal{C}_{1},\ldots,\mathcal{C}_{m-1}$, and thus these codes can also be decoded. The completes the proof of data collection, and also the theorem.

∎

VI Intra-cluster bandwidth for Optimal Codes

We now turn our attention to calculate the amount of intra-cluster repair bandwidth that is needed for a ${(n,k,d)(\alpha,\beta)(m,\ell)}$ code $\mathcal{C}_{m}$ to have optimal file-size $B^{*}$, given by Theorem IV.1. As discussed in Section I-A, there are two contributors to intra-cluster repair bandwidth: $1)$ the local helper bandwidth $\gamma$, which is the amount of data that each of the local helper nodes contributes to repair, and $2)$ the remote helper bandwidth $\gamma^{\prime}$, which is amount of the data that each of the $\ell^{\prime}$ nodes of a remote helper cluster contribute toward computing the $\beta$ symbols of the cluster. In this section, we study individually the minimum requirements on the parameters $\gamma$, $\gamma^{\prime}$ and $\ell^{\prime}$. For obtaining lower bound on $\gamma$, we continue to work with the IFG model in Section III-A, except for the fact that links that connect the local helper out-nodes to the in-node of the replacement node, will have a capacity $\gamma$, instead of $\alpha$. The IFG model in Section III-B will be used when we compute lower bound on $\gamma^{\prime}$ and $\ell^{\prime}$. We also prove the tightness of the bound on $\gamma$ via a converse; however, no such converse is known to us regarding the bound on $\gamma^{\prime}$. We note that, while computing the bound on $\gamma$, we ignore the effects of limited $\gamma^{\prime}$ and $\ell^{\prime}$ (and vice versa), i.e., we assume that $\ell^{\prime}=m$ and $\gamma^{\prime}=\alpha$. Further, the bounds on $\gamma$ and $\gamma^{\prime}$ (or $\ell^{\prime}$) are obtained under the assumptions that $d>0$ and $d\geq k$, respectively.

VI-A Bound on Local Helper Node Intra-cluster Repair Bandwidth, $\gamma$

Theorem VI.1

For an optimal functional repair generalized regenerating code with parameters $\{(n,k,d>0),$ $(\alpha,\beta),(m,\ell)\},\gamma^{\prime}=\alpha,\ell^{\prime}=m$, local helper node bandwidth $\gamma$ is lower-bounded by

[TABLE]

Further, if there is a known upper bound on the number of repairs that occur over the life-time of the system, the above bound is sharp; i.e., the functional repair capacity of the system remains as $B^{*}$ as long as $\gamma\geq\gamma^{*}$.

Proof:

For the bound, we consider a system evolution similar to that used in proof of Theorem IV.1, and demonstrate a cut-set whose value depends on $\gamma$. The lower bound on $\gamma$ follows from the observation that the value of this cut is necessarily lower bounded by $B^{*}$ for a capacity-achieving code. We shall then prove that, as long as $\gamma\geq\gamma^{*}$, the min-cut of any valid IFG is necessarily lower bounded by $B^{*}$; in this case, like in the proof of Theorem IV.1, we know that the functional repair capacity remains as $B^{*}$, as long as there is a known upper bound on the number of repairs in the system. We start with the proof of the lower bound. Consider the same system evolution as in the proof of the bound in Theorem IV.1, except for the $k$-th cluster accessed by the data collector. Thus, physical nodes $X_{i,\ell+1},X_{i,\ell+2},\ldots X_{i,m}$ fail in this order in cluster $i=1$, then in cluster $i=2$, and so on, until cluster $i=k-1$. Note that each of the first $k-1$ clusters experiences a total of $m-\ell$ node failures. For cluster $k$, we consider failure of $m-\ell+1$ nodes, corresponding to physical nodes $X_{k,\ell+1},X_{k,\ell+2},\ldots X_{k,m},X_{k,1}$ in this respective order. In terms of the notation introduced in III-A, the sequence of failures in the $k^{\text{th}}$ cluster correspond to IFG nodes $X_{k,\ell+1}(0),X_{k,\ell+2}(1),\ldots,X_{k,m}(m-\ell-1),X_{k,1}(m-\ell)$. For the repair of $X_{k,\ell+1}(0)$, the local helper nodes used are $X_{k,1}(0),X_{k,2}(0),\ldots,X_{k,\ell}(0)$. For the repair of any of the remaining nodes $X_{k,(\ell+t)\mod(m)+1}(t),1\leq t\leq m-\ell$, the local helper nodes used are $X_{k,2}(t),X_{k,3}(t),\ldots,X_{k,\ell+1}(t)$. Also, clusters $\mathcal{X}_{1}(m-\ell),\mathcal{X}_{2}(m-\ell),\ldots,\mathcal{X}_{\min(d,k-1)}(m-\ell)$ are included in the set of remote clusters that aid in the repair of the $m-\ell+1$ nodes in the $k^{\text{th}}$ cluster. An illustration of the IFG, for the $k^{\text{th}}$ cluster is shown in Fig. 13. Note in this figure that the edges corresponding to local help have capacity $\gamma$.

Let data collector $T$ connect to clusters $\mathcal{X}_{1}(m-\ell),\ldots,\mathcal{X}_{k-1}(m-\ell),\mathcal{X}_{k}(m-\ell+1)$. Consider an $S$-$T$ cut in the IFG that partitions the graph nodes in clusters $1,\cdots,k-1$ in the same way as in IV.1; however it differs in the way the nodes of cluster $k$ are partitioned. The overall set of edges in the cut-set is given as below:

Clusters $1,\ldots,k-1$:

• •

$\{(X_{i,j}^{in}(0)\to X_{i,j}^{out}(0),i\in[k-1],j\in[\ell]\}$. Total capacity of these edges is $(k-1)l\alpha$.

• •

For each $i\in[k-1],t\in[m-\ell]$, either the set of edges $\{(X_{i^{\prime}}^{ext}(0)\to X_{i,\ell+t}^{in}(t)),i^{\prime}\in\{$remote helper cluster indices for the replacement node $X_{i,\ell+t}^{in}(t)\}\backslash[\min\{i-1,d\}]$ or the edge $(X_{i,\ell+t}^{in}(t)\to X_{i,\ell+t}^{out}(t))$. Between the two possibilities, we pick the one which has smaller sum-capacity. In this case, the total capacity of this part of the cut is given by $\sum_{i=1}^{k-1}\sum_{j=\ell+1}^{m}\min\{\alpha,(d-\min\{i-1,d\})\beta\}=(m-\ell)\sum_{i=1}^{k-1}\min\{\alpha,(d-i+1)^{+}\beta\}$.

Cluster $k$:

• •

$(X_{k,1}^{out}(0)\to{X}^{in}_{k,\ell+1}(1))$ of capacity $\gamma$.

• •

$(X_{k,j}^{in}(0)\to X_{k,j}^{out}(0)),\forall j\in[2,\ell]$. Total capacity of these edges is $(\ell-1)\alpha$.

• •

Either the set of edges $\{(X_{i^{\prime}}^{ext}(0)\to X_{k,(\ell+t)\mod(m)+1)}^{in}(t+1)),i^{\prime}\in\{$remote helper cluster indices for the replacement node $X_{k,(\ell+t)\mod(m)+1)}^{in}(t+1)\}\backslash[\min\{i-1,d\}],0\leq t\leq(m-\ell)\}$ or the set of edges $\{(X_{k,(\ell+t)\mod(m)+1)}^{in}(t+1)\to X_{k,(\ell+t)\mod(m)+1)}^{out}(t+1)),0\leq t\leq(m-\ell)\}$. Among the two sets, we pick the one which has smaller sum-capacity. In this case, the total capacity of these edges is $(m-\ell+1)\min\{\alpha,(d-k+1)^{+}\beta\}$.

The value (say, $C_{cut}$) of the cut is given by

[TABLE]

Since we assume an optimal code, it must be true that $C_{cut}\geq B^{*}$, which results in $\gamma\geq\alpha-\min\{\alpha,(d-k+1)^{+}\beta\}$. Finally, note that $\alpha-\min\{\alpha,(d-k+1)^{+}\beta\}=\gamma^{*}$ in the two cases $d<k$, and $d\geq k$. The equivalence for the case $d\geq k$ follows since we only consider $\alpha\geq(d-k+1)\beta$, when $d\geq k$ (see Section IV-A).

We next prove the converse, where we show that, as long as $\gamma\geq\gamma^{*}$, the min-cut of any valid IFG is necessarily lower bounded by $B^{*}$. Toward this, consider the proof of converse part of Theorem IV.1, where we obtained a lower bound on the min-cut of any valid IFG. One can repeat the sequence of arguments exactly as in the proof of converse part of Theorem IV.1, except with the change that the edges corresponding to local help have capacity $\gamma$ (instead of $\alpha$). In this case, it can be seen that instead of (12), we get the following lower bound on min-cut:

[TABLE]

In the above expression, observe that if $\gamma\geq\gamma^{*}$, we have

[TABLE]

whenever $j\leq\ell,i\leq k$. In this case, it follows that (52) can be written as (13). It is then clear that $\text{mincut}(S-T)$ is indeed lower bounded by $B^{*}$ as long as $\gamma\geq\gamma^{*}$. This completes the proof of the converse, and also the theorem. ∎

VI-B Bounds on $\ell^{\prime},\gamma^{\prime}$

In this section, we provide bounds on the parameters $\gamma^{\prime}$ and $\ell^{\prime}$. We use the second IFG model in Section III-B here. Recall that in our setting, for any of the remote helper clusters, we allow any subset of $\ell^{\prime}$ nodes in the cluster to be used to generate the $\beta$ symbols contributed by the cluster. Also, in this section, we make the assumption that the number of remote helper clusters $d\geq k$.

Theorem VI.2

For an optimal functional repair generalized regenerating code with parameters $\{(n,k,d\geq k)$ $(\alpha,\beta),(m,\ell)\},\gamma=\alpha,\ell^{\prime}=m$, the remote helper-node repair bandwidth $\gamma^{\prime}$ is lower-bounded by

[TABLE]

whenever $\alpha\geq(d-k+2)\beta$.

Proof:

We consider data collection from clusters $1$ to $k$. Before data collection, the system experiences $k(m-\ell)$ repairs. Nodes $\ell+1,\ldots,m$ fail and get repaired in cluster $1$ in this respective order. This is followed by failure and repair of nodes $\ell+1,\ldots,m$ in cluster $2$, and so on, until we consider failure and repair of nodes $\ell+1,\ldots,m$ in cluster $k$. In terms of physical nodes, it may be noted that this is the same sequence of failures that was considered in the proof of Theorem IV.1; however, in here, we will impose additional restrictions on the choice of the remote helper clusters. In this proof, external help is taken from the set of the first $d+1$ clusters, excluding the cluster where the failed node resides. Thus, for the repair of $X_{i,j}$, the indices of remote helper clusters are $\{1,\ldots,i-1,i+1,\ldots,k,k+1,\ldots,d+1\}$. The choice of local helper nodes remain same as in the proof of Theorem IV.1, where we used the first $\ell$ nodes in the cluster. An illustration of the IFG is shown in Figure 14.

It can be seen that the following cut-set separates the source from the data collector:

• •

$\{(X_{i,j}^{in}\to X_{i,j}^{out}),i\in[k],j\in[\ell]\}$. Total capacity of these edges is $k\ell\alpha$.

• •

For each $i,1\leq i\leq k$, the edge set with smaller capacity out of $A_{1}(i)\cup A_{2}(i)$ and $A_{3}(i)$ where

• –

$A_{1}(i)\triangleq\{(X_{i^{\prime}}^{ext}\to\widehat{X}_{i,j}^{in}),i^{\prime}\in[k+1,d+1],j\in[\ell+1,m]\}$. Total capacity of edges in $A_{1}(i)$ is $(d-k+1)(m-\ell)\beta$. Recall that $(\widehat{X}_{i,j}^{in},\widehat{X}_{i,j}^{out})$ is simply the replacement node for the failed node $({X}_{i,j}^{in},{X}_{i,j}^{out})$, in the second IFG model that is used here.

• –

$A_{2}(i)\triangleq\{(X_{i^{\prime},j^{\prime}}^{out}\to X_{i^{\prime},X_{i,j}}^{ext}),j\in[\ell+1,m],i^{\prime}\in[i+1,k],j^{\prime}\in[\ell+1,m]\}$. Total capacity of edges in $A_{2}(i)$ is $(m-\ell)(k-i)(m-\ell)\gamma^{\prime}$

• –

$A_{3}(i)\triangleq\{(\widehat{X}_{i,j}^{in}\to\widehat{X}_{i,j}^{out}),j\in[\ell+1,m]\}$. Total capacity of edges in $A_{3}(i)$ is $(m-\ell)\alpha$.

The capacity of the cut-set is given by

[TABLE]

Since we consider optimal codes, we have

[TABLE]

In this case, since we assume that $\alpha\geq(d-k+2)\beta$, we claim that

[TABLE]

To see why (58) is true, if we suppose on the contrary that $(m-\ell)\gamma^{\prime}<\beta$, we have

[TABLE]

The above equation implies that

[TABLE]

where (61) follows from the assumption that $\alpha\geq(d-k+2)\beta$. Clearly, adding up the two corresponding sides (L.HS. and R.H.S.) of (60) and (61), and comparing them contradicts the fact that $C_{cut}\geq B^{*}$. Thus, it must be true that $\gamma^{\prime}\geq\beta/(m-\ell)$, whenever $\alpha\geq(d-k+2)\beta$. ∎

The following theorem establishes the necessary condition on $\ell^{\prime}$ for optimal codes. The proof is along the lines of proof of Theorem VI.2, and is omitted.

Theorem VI.3

For an optimal functional repair GRC with parameters $\{(n,k,d\geq k)$ $(\alpha,\beta),(m,\ell)\},\gamma=\gamma^{\prime}=\alpha$, whenever $\alpha\geq(d-k+2)\beta$, each remote helper cluster must necessarily access all the $m$ nodes in the cluster while generating the $\beta$ symbols; i.e., whenever $\alpha\geq(d-k+2)\beta$, we have $\ell^{\prime}=m$ for an optimal functional repair GRC.

VII Security Under Passive Eavesdropping

In this section, we analyze resilience of the clustered storage system against passive eavesdropping. Our model of clustered storage systems is in part motivated by the need to provide security against an eavesdropper who may gain access to a subset of the clusters. In this context, we extend result in Theorem IV.1 to settings that require security. Below, we first introduce the model for security, and then present the revised file bound. Optimal code construction for security can be provided along the same lines of Construction V.2.

VII-A Passive Eavesdropper Model

The security model is along the lines of the passive eavesdropper model considered in [31], where the authors study security under the classical regenerating code framework. An eavesdropper (say, Eve) gains access to the entire content of any subset of $e$ clusters, where $1\leq e\leq k$. Eve also gets to observe all the helper data that gets downloaded for repair of any node in these $e$ clusters. Eve is passive in the sense that Eve does not change any stored or repair data. The properties of data collection and disk repair remain same as in the case of no eavesdropper (see Section I-A). In this model we $1)$ ignore the effects of intra-cluster bandwidth, and $2)$ restrict ourselves to the setting of deterministic exact repair codes. By deterministic exact repair code, we mean that the helper data for the repair of any node is uniquely determined given the indices of the failed node, local helper nodes and remote helper clusters. We avoid the possibility that the same set of helpers can pass two possible sets of helper data for the repair of the same node.

We wish to store a file such that Eve does not gain any information about it, by having eavesdropped into any subset $E$ of $e$ clusters. To be precise, let $\mathcal{F}^{(s)}$ denote the random variable corresponding to the data file that gets securely stored. We assume the file $\mathcal{F}^{(s)}$ to be uniformly distributed over $\mathbb{F}_{q}^{B^{(s)}}$, and thus $B^{(s)}$ denotes the file-size. We wish to ensure that the mutual information $\mathcal{I}(\mathcal{F}^{(s)};\text{data observed by Eve})=0$. Note that the data observed by Eve not only includes the content of the $E$ clusters, but also any inter-cluster helper data that is received toward the repair of nodes in these clusters. We shall write $\mathcal{C}^{(s)}_{m}$ to denote a secure generalized regenerating code, and its parameter set will be identified with $\{(n,k,d),(\alpha,\beta),(m,\ell),(e)$}.

VII-B File Size Under Exact Repair

In this section, we obtain an upper bound on the file-size $B^{(s)}$ of the exact repair secure generalized regenerating code $\mathcal{C}^{(s)}_{m}$. To derive the bound, we use information theoretic techniques similar to those used in [36], [31]. We begin with some necessary notation. Let $Y_{i,j}\in\mathbb{F}_{q}^{\alpha},1\leq i\leq n,1\leq j\leq m$ denote the content stored in node $j$ of cluster $i$. We write $\bf{Y}_{i}$ to denote $[Y_{i,1}\ldots Y_{i,m}],1\leq i\leq n$. The property of data collection requires that

[TABLE]

where $H(.)$ denotes the entropy function computed with respect to $\log q$. Next, consider the repair of node $j$ in cluster $i$. Let $\mathcal{H}\subset[n]\backslash\{i\},|\mathcal{H}|=d$, and $\mathcal{L}\subset[m]\backslash\{j\},|\mathcal{L}|=\ell$ respectively denote the indices of remote helper clusters and local helper nodes that aid in the repair process. Let $Z_{i^{\prime},i,j}^{\mathcal{H},\mathcal{L}}$ denote helper data passed by cluster $i^{\prime}$. Recall our assumption that the exact repair code is deterministic, and thus $Z_{i^{\prime},i,j}^{\mathcal{H},\mathcal{L}}$ is uniquely determined as a function of $(i,j),\{Y_{i,j^{\prime}},j^{\prime}\in\mathcal{L}\}$ and $\{Y_{i^{\prime}},i^{\prime}\in\mathcal{H}\}$. The property of exact repair is jointly characterized by the following set of inequalities:

[TABLE]

Next, define $U_{i}$ to denote the collection of all the inter-cluster helper data ever received toward the repair of nodes in cluster $i$, i.e.,

[TABLE]

The property of being secure against the passive eavesdropper Eve is equivalent to saying that

[TABLE]

The following theorem characterizes the file-size bound under passive eavesdropping.

Theorem VII.1

The file-size of a secure exact-repair deterministic generalized regenerating code having parameters $\{(n,k,d),(\alpha,\beta),(m,\ell),(e)\}$ is upper bounded by

[TABLE]

Proof:

Without loss of generality, let us assume that $E=\{1,2,\ldots,e\}$. Using (67), the file-size $B^{(s)}$ is given by

[TABLE]

From (62), we further get that

[TABLE]

Combining (69) and (70), we get

[TABLE]

where (76) follows from (63)-(65). This completes the proof of the upper bound. ∎

As before, whenever $d>0$, we associate the operating point $\alpha=d\beta$ with the minimum repair bandwidth secure regenerating codes. Construction V.2 can be easily adapted to construct optimal secure exact repair codes at the MBR point. In the modified construction, one combines $\ell$ secure MDS codes for the wiretap-II channel [32][33], and $(m-\ell)$ classical exact-repair secure MBR codes [31][34]. The construction and proof of optimality are similar to Construction V.2; we avoid a full description here. A pictorial illustration of the secure code construction appears in Fig. 15. Finally, we note that the bound in (77) is weak at the MSR point, since it is known that classical exact-repair secure MSR codes in general cannot achieve file size $\sum_{i=e}^{k-1}\min\{\alpha,(d-i)^{+}\beta\}=(k-e)\alpha$ [37, 38]. It is an interesting question as to whether the analysis like in [37, 38] can be used to provide a tighter bound for secure MSR codes under the framework of GRCs considered in this paper.

VIII Conclusions

To conclude, we study the problem of storage-overhead vs repair-bandwidth overhead in clustered storage systems. The notion of clustering is used in both data collection and node repair. For data collection, we require retrievability using content from any set of $k$ clusters. For node repair, we take the help of surviving local nodes in the host cluster, as well as from other remote clusters. We first characterize the optimal file-size that is achievable while ignoring intra-cluster bandwidth costs, and then obtain bounds on intra-cluster bandwidth costs that is needed to achieve this file-size. Our results show that while it is beneficial to increase the number of local helper nodes ($\ell$) during repair in order to simultaneously improve both storage and inter-cluster bandwidth costs, increasing $\ell$ has an adverse effect on intra-cluster repair bandwidth. Our bounds on file-size and intra-cluster bandwidth give guidelines for choosing the desired number of local helper nodes in practice, based on the relative costs of the various metrics. We present constructions of optimal exact and functional repair codes, which enable operating points for clustered systems that are not achievable via previously known coding solutions. We also analyze the resilience of the system against passive eavesdropping.

Two key questions remain at the end of this work. Firstly, our bounds on intra-cluster bandwidth are derived under the assumption of functional repair. It is unclear if these bounds hold under exact repair; specifically at the minimum-inter-cluster bandwidth operating point. The exact repair constructions in this paper, although they have optimal file-size (and inter-cluster bandwidth), incur the maximum possible intra-cluster bandwidth. Secondly, the bound on any one of the intra-cluster bandwidth related parameters (say, $\gamma$) was derived without limiting the other two other parameters ($\gamma^{\prime},\ell^{\prime}$). It is of special interest to know the simultaneously optimality of the the bounds in (2) and (3). We believe that a first step in this direction would be to prove a converse statement (achievability) to (3). Achievability of (3) is indeed suggested by RLNC-based simulations.

Finally, model extensions of interest include the case of simultaneous recovery from multiple node failures in a given cluster. While studying recovery for multiple node failures, it is of interest to consider presence of local parity relations in each cluster, even when $d>0$.