Robust Routing Made Easy

TL;DR

This paper introduces simple, generic blackbox transformations that enhance the robustness of existing routing schemes against independent node failures, maintaining network connectivity with minimal resource overhead.

Contribution

It presents a novel method to reinforce existing networks against failures using blackbox transformations, simplifying design and verification of robust routing.

Findings

Transforms increase resilience against independent failures

Minimal additional resources needed for reinforcement

Effective in real-world network topologies

Abstract

With the increasing scale of communication networks, the likelihood of failures grows as well. Since these networks form a critical backbone of our digital society, it is important that they rely on robust routing algorithms which ensure connectivity despite such failures. While most modern communication networks feature robust routing mechanisms, these mechanisms are often fairly complex to design and verify, as they need to account for the effects of failures and rerouting on communication. This paper revisits the design of robust routing mechanisms, with the aim to avoid such complexity. In particular, we showcase simple and generic blackbox transformations that increase resilience of routing against independently distributed failures, which allows to simulate the routing scheme on the original network, even in the presence of non-benign node failures (henceforth called faults). This is attractive as the system specification and routing policy can simply be preserved. We present a scheme for constructing such a reinforced network, given an existing (synchronous) network and a routing scheme. We prove that this algorithm comes with small constant overheads, and only requires a minimal amount of additional node and edge resources. At the same time, it allows to tolerate a large number of independent random (node) faults, asymptotically almost surely. We complement our analytical results with simulations on different real-world topologies.

Equations

$$1-n\sum _{j=f+1}^{2f+1}\left(\genfrac{}{}{0ex}{}{2f+1}{j}\right)p^j(1-p{)}^{2f+1-j}$$

$$\ge$$

$$\left(\genfrac{}{}{0ex}{}{2f+1}{f+1}\right)p^{f+1}(1-p{)}^f\ge {\left(\frac{3}{2}\right)}^f{p}^{f+1}(1-p{)}^f\ge p^{f+1},$$

$$1-{\left(1-p^{f+1}\right)}^{\Omega (n)}\subseteq 1-{\left(1-\omega \left(\frac{1}{n}\right)\right)}^{\Omega (n)}=1-o(1).$$

$$\mathrm{Pr}\left[\{v_i|i\in [\ell ]\}\cap F^{\prime }=\ell \right]=p^{f+1}.$$

$$\displaystyle E_{e}^{\prime}\triangleq\begin{cases}\{(v_{i},w_{i})\,|\,i\in[\ell]\}&\mbox{if $\exists k^{\prime}\in[k]:v,w\in R_{k^{\prime}}$}\\ \{(v_{i},w_{j})\,|\,i,j\in[\ell]\}&\mbox{else,}\end{cases}$$

$$\mathrm{Pr}\left[\{v_i|v\in R_{k^{\prime }}\}\cap F^{\prime }=\varnothing \right]=(1-p{)}^{|R_{k^{\prime }}|}\ge 1-Rp.$$

$$\sum _{j=f+1}^{2f+1}\left(\genfrac{}{}{0ex}{}{2f+1}{j}\right)(Rp{)}^j(1-Rp{)}^{2f+1-j}$$

$$=$$

$$\mathrm{Pr}\left[|\{v_i\}\cap F^{\prime }|\ge f+1\right]\ge p^{f+1}=\omega \left(\frac{1}{n}\right).$$

$$\frac{p_k}{p_1}=\frac{p_{k}n}{p_{1}n}\approx \frac{0.01^{1/k}}{0.01}=100^{1-1/k}.$$

Full text

This paper revisits the design of robust routing mechanisms, with the aim to avoid such complexity. In particular, we showcase simple and generic blackbox transformations that increase resilience of routing against independently distributed failures, which allows to simulate the routing scheme on the original network, even in the presence of non-benign node failures (henceforth called faults). This is attractive as the system specification and routing policy can simply be preserved.

We present a scheme for constructing such a reinforced network, given an existing (synchronous) network and a routing scheme. We prove that this algorithm comes with small constant overheads, and only requires a minimal amount of additional node and edge resources. At the same time, it allows to tolerate a large number of independent random (node) faults, asymptotically almost surely. We complement our analytical results with simulations on different real-world topologies.

I Introduction

Communication networks have become a critical backbone of our digital society. For example, many datacentric applications related to entertainment, social networking, or health, among others, are distributed and rely on the high availability and dependability of the interconnecting network (e.g., a datacenter network or a wide-area network). At the same time, with the increasing scale of today’s distributed and networked systems (often relying on commodity hardware as a design choice [2, 7, 25]), the number of failures is likely to increase as well [26, 27, 31, 44, 45]. It is hence important that communication networks can tolerate such failures and remain operational despite the failure of some of their components.

Robust routing mechanisms aim to provide such guarantees: by rerouting traffic quickly upon failures, reachability is preserved. Most communication networks readily feature robust routing mechanisms, in the control plane (e.g. [23, 24, 28, 47]), in the data plane (e.g. [6, 32, 43, 48]), as well as on higher layers (e.g. [4]). However, the design of such robust routing mechanisms is still challenging and comes with tradeoffs, especially if resilience should extend to multiple failures [12].

Besides a fast reaction time and re-establishing connectivity, the resulting routes typically need to fulfill certain additional properties, related to the network specification and policy. Ensuring such properties however can be fairly complex, as packets inevitably follow different paths after failures. Interestingly, while the problem of how to re-establish reachability after failures is well explored, the problem of providing specific properties on the failover paths is much less understood.

This paper revisits the design of robust routing mechanisms and presents a new approach to robust routing which conceptually differs significantly from existing literaure. In particular, our approach aims to overcome the complexities involved in designing robust routing algorithms, by simply sticking to the original network and routing specification. To achieve this, our approach is to mask the effects of failures using redundancy: in the spirit of error correction, we proactively reinforce networks by adding a minimal number of additional nodes and links, rather than coping with failed components when they occur. The latter is crucial for practicability: significant refactoring of existing systems and/or accommodating substantial design constraints is rarely affordable.

In this paper, to ensure robustness while maintaining the network and routing specification, we aim to provide a high degree of fault-tolerance, which goes beyond simple equipment and failstop failures, but accounts for more general faults which include non-benign failures of entire nodes.

While our approach presented in this paper will be general and applies to any network topology, we are particularly interested in datacenter networks (e.g., based on low-dimensional hypercubes or $d$-dimensional tori [29, 40]) as well as in wide-area networks (which are typically sparse [58]). We will show that our approach works especially well for these networks.

I-A The Challenge

More specifically, we are given a network $G=(V,E)$ and a routing scheme, i.e., a set of routes in $G$. We seek to reinforce the network $G$ by allocating additional resources, in terms of nodes and edges, and to provide a corresponding routing strategy to simulate the routing scheme on the original network despite non-benign node failures.

The main goal is to maximize the probability that the network withstands failures (in particular, random failures of entire nodes), while minimizing the resource overhead. Furthermore, we want to ensure that the network transformation is simple to implement, and that it interferes as little as possible with the existing system design and operation, e.g., it does not change the reinforced system’s specification.

Toward this goal, in this paper, we make a number of simplifying assumptions. First and most notably, we assume independent failures, that is, we aim at masking faults with little or no correlation among each other. Theoretically, this is motivated by the fact that guaranteeing full functionality despite having $f$ adversarially placed faults trivially requires redundancy (e.g., node degrees) larger than $f$. There is also practical motivation to consider independent faults: many distributed systems proactively avoid fault clusters [19, 57] and there is also empirical evidence that in certain scenarios, failures are only weakly correlated [39].

Second, we treat nodes and their outgoing links as fault-containment regions (according to [36]), i.e., they are the basic components our systems are comprised of. This choice is made for the sake of concreteness; similar results could be obtained when considering, e.g., edge failures, without changing the gist of results or techniques. With these considerations in mind, the probability of uniformly random node failures that the reinforced system can tolerate is a canonical choice for measuring resilience.

Third, we focus on synchronous networks, for several reasons: synchrony not only helps in handling faults, both on the theoretical level (as illustrated by the famous FLP theorem [20]) and for ensuring correct implementation, but it also simplifies presentation, making it easier to focus on the proposed concepts. In this sense, we believe that our approach is of particular interest in the context of real-time systems, where the requirement of meeting hard deadlines makes synchrony an especially attractive choice.

I-B Contributions and Techniques

This paper proposes a novel and simple approach to robust routing, which decouples the task of designing a reinforced network from the task of designing a routing scheme over the input network. By virtue of this decoupling, our approach supports arbitrary routing schemes and objectives, from load minimization to throughput maximization and beyond, in various models of computation, e.g., centralized or distributed, randomized or deterministic, online or offline, or oblivious.

We first consider a trivial approach: we simply replace each node by $\ell\in{\mathbb{N}}$ copies and for each edge we connect each pair of copies of its endpoints, where $\ell$ is a constant.111Choosing concreteness over generality, we focus on the, in our view, most interesting case of constant $\ell$. It is straightforward to generalize the analysis. Whenever a message would be sent over an edge in the original graph, it should be sent over each copy of the edge in the reinforced graph. If not too many copies of a given node fail, this enables each receiving copy to recover the correct message. Thus, each non-faulty copy of a node can run the routing algorithm as if it were the original node, guaranteeing that it has the same view of the system state as its original in the corresponding fault-free execution of the routing scheme on the original graph.

When analyzing this approach, we observe that asymptotically almost surely (a.a.s., with probability $1-o(1)$) and with $\ell=2f+1$, this reinforcement can sustain an independent probability $p$ of $f$ Byzantine node failures222Byzantine [50] nodes may violate the protocol in any way, including collusion. for any $p\in o(n^{-1/(f+1)})$. This threshold is sharp up to (small) constant factors: for $p\in\omega(n^{-1/(f+1)})$, a.a.s. there is some node for which all of its copies fail. If we restrict the fault model to omission faults (faulty nodes may skip sending some messages), $\ell=f+1$ suffices. The cost of this reinforcement is that the number of nodes and edges increase by factors of $\ell$ and $\ell^{2}$, respectively. Therefore, already this simplistic solution can support non-crash faults of probability $p\in o(1/\sqrt{n})$ at a factor-$4$ overhead.

We note that the simulation introduces no large computational overhead and does not change the way the system works, enabling to use it as a blackbox. Also randomized algorithms can be simulated in a similar fashion, provided that all copies of a node have access to a shared source of randomness. Note that this requirement is much weaker than globally shared randomness: it makes sense to place the copies of a node in physical proximity to approximately preserve the geometrical layout of the physical realization of the network topology.

Our approach above raises the question whether we can reduce the involved overhead further. In this paper, we will answer this question positively: We propose to apply the above strategy only to a small subset $E^{\prime}$ of the edge set. Denoting by $v_{1},\ldots,v_{\ell}$ the copies of node $v\in V$, for any remaining edge $\{v,w\}\in E\setminus E^{\prime}$ we add only edges $\{v_{i},w_{i}\}$, $i\in[\ell]$, to the reinforced graph. The idea is to choose $E^{\prime}$ in a way such that the connected components induced by $E\setminus E^{\prime}$ are of constant size, yet $|E^{\prime}|=\varepsilon|E|$. This results in the same asymptotic threshold for $p$, while the number of edges of the reinforced graph drops to $((1-\varepsilon)\ell+\varepsilon\ell^{2})|E|$. For any constant choice of $\varepsilon$, we give constructions with this property for grids or tori of constant dimension and minor-free graphs of bounded degree. Again, we consider the case of $f=1$ of particular interest: in many typical network topologies, we can reinforce the network to boost the failure probability that can be tolerated from $\Theta(1/n)$ to $\Omega(1/\sqrt{n})$ by roughly doubling (omission faults) or tripling (Byzantine faults) the number of nodes and edges.

The redundancy in this second construction is near-optimal under the constraint that we want to simulate an arbitrary routing scheme in a blackbox fashion, as it entails that we need a surviving copy of each edge, and thus in particular each node. In many cases, the paid price will be smaller than the price for making each individual component sufficiently reliable to avoid this overhead. Furthermore, we will argue that the simplicity of our constructions enables us to re-purpose the redundant resources in applications with less strict reliability requirements.

Our results show that while approach is general and can be applied to any existing network topology (we will describe and analyze valid reinforcements for our faults models on general graphs), it can be refined and is particularly interesting in the context of networks that admit suitable partitionings. Such networks include sparse, minor-free graphs, which are practically relevant topologies in wide-area networks, as well as torus graphs and low-dimensional hypercubes, which arise in datacenters and parallel architectures.

To complement our theoretical findings and investigate the reinforcement cost in real networks, we conducted experiments on the Internet Topology Zoo [35]. We find that our approach achieves robustness at significantly lower cost compared to the naive replication strategy often employed in dependable networks.

I-C Organization

In § II, we sketch the properties of our approach and state a number of potential applications. In § III, we formalize the fault models that we tackle in this article alongside the notion of a valid reinforcement and its complexity measures. In § IV and § V, we study valid reinforcements on general graphs, and in § VI, we study more efficient reinforcements for specific graphs. We complement our analytical results with an empirical simulation study in § VII. In § VIII we raise a number of points in favor of the reinforcement approach. We review related work in § IX, and we conclude and present a number of interesting follow-up questions in § X.

II High-level Overview: Reinforcing Networks

Let us first give an informal overview of our blackbox transformation for reinforcing networks (for formal specification see § III), as well as its guarantees and preconditions.

Assumptions on the Input Network

We have two main assumptions on the network at hand: (1) We consider synchronous routing networks, and (2) each node in the network (alongside its outgoing links) is a fault-containment region, i.e., it fails independently from other nodes. We do not make any assumptions on the network topology, but will provide specific optimizations for practically relevant topologies (such as sparse, minor-free networks or hypercubes) in § VI.

Valid Reinforcement Simulation Guarantees

Our reinforcements create a number of copies of each node. We have each non-faulty copy of a node run the routing algorithm as if it were the original node, guaranteeing that it has the same view of the system state as its original in the corresponding fault-free execution of the routing scheme on the original graph. Moreover, the simulation fully preserves all guarantees of the schedule, including its timing, and introduces no big computational overhead. This assumption is simple to meet in stateless networks, while it requires synchronization primitives in case of stateful network functions.

Unaffected Complexity and Cost Measures

Routing schemes usually revolve around objective functions such as load minimization, maximizing the throughput, minimizing the latency, etc., while aiming to minimize complexity related to, e.g., the running time for centralized algorithms, the number of rounds for distributed algorithms, the message size, etc. Moreover, there is the degree of uncertainty that can be sustained, e.g., whether the input to the algorithm is fully available at the beginning of the computation (offline computation) or revealed over time (online computation). Our reinforcements preserve all of these properties, as they operate in a blackbox fashion. For example, our machinery readily yields various fault-tolerant packet routing algorithms in the Synchronous Store-and-Forward model by Aiello et. al [1]. More specifically, from [16] we obtain a centralized deterministic online algorithm on unidirectional grids of constant dimension that achieves a competitive ratio which is polylogarithmic in the number of nodes of the input network w.r.t. throughput maximization. Using [17] instead, we get a centralized randomized offline algorithm on the unidirectional line with constant approximation ratio w.r.t. throughput maximization. In the case that deadlines need to be met the approximation ratio is, roughly, $O(\log^{*}n)$ [55]. As a final example, one can obtain from [5] various online distributed algorithms with sublinear competitive ratios w.r.t. throughput maximization.

Cost and Gains of the Reinforcement

The price of adding fault-tolerance is given by the increase in the network size, i.e., the number of nodes and edges of the reinforced network in comparison to the original one. Due to the assumed independence of node failures, it is straightforward to see that the (uniform) probability of sustainable node faults increases roughly like $n^{-1/(f+1)}$ in return for (i) a linear-in-$f$ increase in the number of nodes and (ii) an increase in the number of edges that is quadratic in $f$. We then proceed to improve the construction for grids and minor-free constant-degree graphs to reduce the increase in the number of edges to being roughly linear in $f$. Based on this information, one can then assess the effort in terms of these additional resources that is beneficial, as less reliable nodes in turn are cheaper to build, maintain, and operate. We also note that, due to the ability of the reinforced network to ensure ongoing unrestricted operability in the presence of some faulty nodes, faulty nodes can be replaced or repaired before communication is impaired or breaks down.

Preprocessing

Preprocessing is used, e.g., in computing routing tables in Oblivious Routing [54, 56]. The reinforcement simply uses the output of such a preprocessing stage in the same manner as the original algorithm. In other words, the preprocessing is done on the input network and its output determines the input routing scheme. In particular, the preprocessing may be randomized and does not need to be modified in any way.

Randomization

Randomized routing algorithms can be simulated as well, provided that all copies of a node have access to a shared source of randomness. We remark that, as our scheme locally duplicates the network topology, it is natural to preserve the physical realization of the network topology in the sense that all (non-faulty) copies of a node are placed in physical proximity. This implies that this constraint is much easier to satisfy than globally shared randomness.

III Preliminaries

We consider synchronous routing networks. Formally, the network is modeled as a directed graph $G=(V,E)$, where $V$ is the set of $n\triangleq|V|$ vertices, and $E$ is the set of $m\triangleq|E|$ edges (or links). Each node maintains a state, based on which it decides in each round for each of its outgoing links which message to transmit. We are not concerned with the inner workings of the node, i.e., how the state is updated; rather, we assume that we are given a scheduling algorithm performing the task of updating this state and use it in our blackbox transformations. In particular, we allow for online, distributed, and randomized algorithms.

Probability-$p$ Byzantine Faults Byz$\left(p\right)$

The set of faulty nodes $F\subseteq V$ is determined by sampling each $v\in V$ into $F$ with independent probability $p$. Nodes in $F$ may deviate from the protocol in arbitrary ways, including delaying, dropping, or forging messages, etc.

Probability-$p$ Omission Faults Om$\left(p\right)$

The set of faulty nodes $F\subseteq V$ is determined by sampling each $v\in V$ into $F$ with independent probability $p$. Nodes in $F$ may deviate from the protocol by not sending a message over an outgoing link when they should. We note that it is sufficient for this fault model to be satisfied logically. That is, as long as a correct node can identify incorrect messages, it may simply drop them, resulting in the same behavior of the system at all correct nodes as if the message was never sent.

Simulations and Reinforcement

For a given network $G=(V,E)$ and a scheduling algorithm $A$, we will seek to reinforce $(G,A)$ by constructing $G^{\prime}=(V^{\prime},E^{\prime})$ and scheduling algorithm $A^{\prime}$ such that the original algorithm $A$ is simulated by $A^{\prime}$ on $G^{\prime}$, where $G^{\prime}$ is subject to random node failures. We now formalize these notions. First, we require that there is a surjective mapping $P:V^{\prime}\to V$; fix $G^{\prime}$ and $P$, and choose $F^{\prime}\subseteq V^{\prime}$ randomly as specified above.

Definition 1** (Simulation under Byz$\left(p\right)$).**

Assume that in each round $r\in{\mathbb{N}}$, each $v^{\prime}\in V^{\prime}\setminus F^{\prime}$ is given the same input by the environment as $P(v^{\prime})$. $A^{\prime}$ is a simulation of $A$ under Byz$\left(p\right)$, if for each $v\in V$, a strict majority of the nodes $v^{\prime}\in V^{\prime}$ with $P(v^{\prime})=v$ computes in each round $r\in{\mathbb{N}}$ the state of $v$ in $A$ in this round. The simulation is strong, if not only for each $v\in V$ there is a strict majority doing so, but all $v^{\prime}\in V^{\prime}\setminus F^{\prime}$ compute the state of $P(v^{\prime})$ in each round.

Definition 2** (Simulation under Om$\left(p\right)$).**

Assume that in each round $r\in{\mathbb{N}}$, each $v^{\prime}\in V^{\prime}$ is given the same input by the environment as $P(v^{\prime})$. $A^{\prime}$ is a simulation of $A$ under Om$\left(p\right)$, if for each $v\in V$, there is $v^{\prime}\in V^{\prime}$ with $P(v^{\prime})=v$ that computes in each round $r\in{\mathbb{N}}$ the state of $v$ in $A$ in this round. The simulation is strong, if each $v^{\prime}\in V^{\prime}$ computes the state of $P(v^{\prime})$ in each round.

Definition 3** (Reinforcement).**

A (strong) reinforcement of a graph $G=(V,E)$ is a graph $G^{\prime}=(V^{\prime},E^{\prime})$, a surjective mapping $P\colon V^{\prime}\to V$, and a way of determining a scheduling algorithm $A^{\prime}$ for $G^{\prime}$ out of scheduling algorithm $A$ for $G$. The reinforcement is valid under the given fault model (Byz$\left(p\right)$ or Om$\left(p\right)$) if $A^{\prime}$ is a (strong) simulation of $A$ a.a.s.

Resources and Performance Measures.

We use the following performance measures.

1. (i)

The probability $p$ of independent node failures that can be sustained a.a.s. 2. (ii)

The ratio $\nu\triangleq|V^{\prime}|/|V|$, i.e., the relative increase in the number of nodes. 3. (iii)

The ratio $\eta\triangleq|E^{\prime}|/|E|$, i.e., the relative increase in the number of edges.

We now briefly discuss, from a practical point of view, why we do not explicitly consider further metrics that are of interest.

Other Performance Measures

• •

Latency: As our reinforcements require (time-preserving) simulation relations, in terms of rounds, there is no increase in latency whatsoever. However, we note that (i) we require all copies of a node to have access to the input (i.e., routing requests) of the simulated node and (ii) our simulations require to map received messages in $G^{\prime}$ to received messages of the simulated node in $G$. Regarding (i), recall that it is beneficial to place all copies of a node in physical vicinity, implying that the induced additional latency is small. Moreover, our constructions naturally lend themselves to support redundancy in computations as well, by having each copy of a node perform the tasks of its original; in this case, (i) comes for free. Concerning (ii), we remark that the respective operations are extremely simple; implementing them directly in hardware is straightforward and will have limited impact on latency in most systems.

• •

Bandwidth/link capacities. We consider the uniform setting in this work. Taking into account how our simulations operate, one may use the ratio $\eta$ as a proxy for this value.

• •

Energy consumption. Regarding the energy consumption of links, the same applies as for bandwidth. The energy nodes use for routing computations is the same as in the original system, except for the overhead induced by Point (ii) we discussed for latency. Neglecting the latter, the energy overhead is in the range $[\min\{\nu,\eta\},\max\{\nu,\eta\}]$.

• •

Hardware cost. Again, neglecting the computational overhead of the simulation, the relative overhead lies in the range $[\min\{\nu,\eta\},\max\{\nu,\eta\}]$

In light of these considerations, we focus on $p$, $\nu$, and $\eta$ as key metrics for evaluating the performance of our reinforcement strategies.

IV Strong Reinforcement under Byz$\left(p\right)$

We now present and analyze valid reinforcements under Byz$\left(p\right)$ for our faults model on general graphs. Given are the input network $G=(V,E)$ and scheduling algorithm $A$. Fix a parameter $f\in{\mathbb{N}}$ and set $\ell=2f+1$.

Reinforced Network $G^{\prime}$

We set $V^{\prime}\triangleq V\times[\ell]$, where $[\ell]\triangleq\{1,\ldots,\ell\}$, and denote $v_{i}\triangleq(v,i)$. Accordingly, $P(v_{i})\triangleq v$. We define $E^{\prime}\triangleq\{(v^{\prime},w^{\prime})\in V^{\prime}\times V^{\prime}\,|\,(P(v^{\prime}),P(w^{\prime}))\in E\}$.

Strong Simulation $A^{\prime}$ of $A$

Consider node $v^{\prime}\in V^{\prime}\setminus F^{\prime}$. We want to maintain the invariant that in each round, each such node has a copy of the state of $v=P(v^{\prime})$ in $A$. To this end, $v^{\prime}$

1. (1)

initializes local copies of all state variables of $v$ as in $A$, 2. (2)

sends on each link $(v^{\prime},w^{\prime})\in E^{\prime}$ in each round the message $v$ would send on $(P(v^{\prime}),P(w^{\prime}))$ when executing $A$, and 3. (3)

for each neighbor $w$ of $P(v^{\prime})$ and each round $r$, updates the local copy of the state of $A$ as if $v$ received the message that has been sent to $v^{\prime}$ by at least $f+1$ of the nodes $w^{\prime}$ with $P(w^{\prime})=w$ (each one using edge $(w^{\prime},v^{\prime})$).

Naturally, the last step requires such a majority to exist; otherwise, the simulation fails. We show that $A^{\prime}$ can be executed and simulates $A$ provided that for each $v\in V$, no more than $f$ of its copies are in $F^{\prime}$.

Lemma 1**.**

If for each $v\in V$, $|\{v_{i}\in F^{\prime}\}|\leq f$, then $A^{\prime}$ strongly simulates $A$.

Proof.

We show the claim by induction on the round number $r\in{\mathbb{N}}$, where we consider the initialization to anchor the induction at $r=0$. For the step from $r$ to $r+1$, observe that because all $v^{\prime}\in V^{\prime}\setminus F^{\prime}$ have a copy of the state of $P(v^{\prime})$ at the end of round $r$ by the induction hypothesis, each of them can correctly determine the message $P(v^{\prime})$ would send over link $(v,w)\in E$ in round $r+1$ and send it over each $(v^{\prime},w^{\prime})\in E$ with $P(w^{\prime})=w$. Accordingly, each $v^{\prime}\in V^{\prime}\setminus F^{\prime}$ receives the message $A$ would send over $(w,v)\in E$ from each $w^{\prime}\in V^{\prime}\setminus F^{\prime}$ with $P(w^{\prime})=w$ (via the link $(w^{\prime},v^{\prime})$). By the assumption of the lemma, we have at least $\ell-f=f+1$ such nodes, implying that $v^{\prime}$ updates the local copy of the state of $A$ as if it received the same messages as when executing $A$ in round $r+1$. Thus, the induction step succeeds and the proof is complete. ∎

Resilience of the Reinforcement

We now examine how large the probability $p$ can be for the precondition of Lemma 1 to be satisfied a.a.s.

Theorem 1**.**

If $p\in o(n^{-1/(f+1)})$, the above construction is a valid strong reinforcement for the fault model Byz$\left(p\right)$. If $G$ contains $\Omega(n)$ nodes with non-zero outdegree, $p\in\omega(n^{-1/(f+1)})$ implies that the reinforcement is not valid.

Proof.

By Lemma 1, $A^{\prime}$ strongly simulates $A$ if for each $v\in V$, $|\{v_{i}\in F^{\prime}\}|\leq f$. If $p\in o(n^{-1/(f+1)})$, using $\ell=2f+1$ and a union bound we see that the probability of this event is at least

[TABLE]

Here, the second to last step uses that $\binom{a}{b}\leq(ae/b)^{b}$ and the final step exploits that $p\in o(n^{-1/(f+1)})$.

For the second claim, assume w.l.o.g. $p\leq 1/3$, as increasing $p$ further certainly increases the probability of the system to fail. For any $v\in V$, the probability that $|\{v_{i}\in F^{\prime}\}|>f$ is independent of the same event for other nodes and larger than

[TABLE]

since $\binom{a}{b}\geq(a/b)^{b}$ and $1-p\geq 2/3$. Hence, if $G$ contains $\Omega(n)$ nodes $v$ with non-zero outdegree, $p\in\omega(n^{-1/(f+1)})$ implies that the probability that there is such a node $v$ for which $|\{v_{i}\in F^{\prime}\}|>f$ is at least

[TABLE]

If there is such a node $v$, there are algorithms $A$ and inputs so that $A$ sends a message across some edge $(v,w)$ in some round. If faulty nodes do not send messages in this round, the nodes $w_{i}\in V^{\prime}\setminus F^{\prime}$ do not receive the correct message from more than $f$ nodes $v_{i}$ and the simulation fails. Hence, the reinforcement cannot be valid. ∎

Remark 1**.**

For constant $p$, one can determine suitable values of $f\in\Theta(\log n)$ using Chernoff’s bound. However, as our focus is on small (constant) overhead factors, we refrain from presenting the calculation here.

Efficiency of the Reinforcement

For $f\in{\mathbb{N}}$, we have that $\nu=\ell=2f+1$ and $\eta=\ell^{2}=4f^{2}+4f+1$, while we can sustain $p\in o(n^{-1/(f+1)})$. In the special case of $f=1$, we improve from $p\in o(1/n)$ for the original network to $p\in o(1/\sqrt{n})$ by tripling the number of nodes. However, $\eta=9$, i.e., while the number of edges also increases only by a constant, it seems too large in systems where the limiting factor is the amount of links that can be afforded.

V Strong Reinforcement under Om$\left(p\right)$

The strong reinforcement from the previous section is, trivially, also a strong reinforcement under Om$\left(p\right)$. However, we can reduce the number of copies per node for the weaker fault model. Given are the input network $G=(V,E)$ and scheduling algorithm $A$. Fix a parameter $f\in{\mathbb{N}}$ and, this time, set $\ell=f+1$.

Reinforced Network $G^{\prime}$

We set $V^{\prime}\triangleq V\times[\ell]$ and denote $v_{i}\triangleq(v,i)$. Accordingly, $P(v_{i})\triangleq v$. We define $E^{\prime}\triangleq\{(v^{\prime},w^{\prime})\in V^{\prime}\times V^{\prime}\,|\,(P(v^{\prime}),P(w^{\prime}))\in E\}$.

Strong Simulation $A^{\prime}$ of $A$

Each node333Nodes suffering omission failures still can simulate $A$ correctly. $v^{\prime}\in V^{\prime}$

1. (1)

initializes local copies of all state variables of $v$ as in $A$, 2. (2)

sends on each link $(v^{\prime},w^{\prime})\in E^{\prime}$ in each round the message $v$ would send on $(P(v^{\prime}),P(w^{\prime}))$ when executing $A$, and 3. (3)

for each neighbor $w$ of $P(v^{\prime})$ and each round $r$, updates the local copy of the state of $A$ as if $v$ received the (unique) message that has been sent to $v^{\prime}$ by some of the nodes $w^{\prime}$ with $P(w^{\prime})=w$ (each one using edge $(w^{\prime},v^{\prime})$).

Naturally, the last step assumes that some such neighbor sends a message and all $w^{\prime}$ with $P(w^{\prime})$ send the same such message; otherwise, the simulation fails. We show that $A^{\prime}$ can be executed and simulates $A$ provided that for each $v\in V$, no more than $f$ of its copies are in $F^{\prime}$.

Lemma 2**.**

If for each $v\in V$, $|\{v_{i}\in F^{\prime}\}|\leq f$, $A^{\prime}$ strongly simulates $A$.

Proof.

Analogous to the one of Lemma 1, with the difference that faulty nodes may only omit sending messages and thus a single correct copy per node is sufficient. ∎

Resilience of the Reinforcement

We now examine how large the probability $p$ can be for the precondition of Lemma 1 to be satisfied a.a.s.

Theorem 2**.**

The above construction is a valid strong reinforcement for the fault model Om$\left(p\right)$ if $p\in o(n^{-1/(f+1)})$. If $G$ contains $\Omega(n)$ nodes with non-zero outdegree, $p\in\omega(n^{-1/(f+1)})$ implies that the reinforcement is not valid.

Proof.

By Lemma 2, $A^{\prime}$ strongly simulates $A$ if for each $v\in V$, $|\{v_{i}\in F^{\prime}\}|\leq f=\ell-1$. For $v\in V$,

[TABLE]

By a union bound, $A^{\prime}$ thus simulates $A$ with probability $1-o(1)$ if $p\in o(n^{-1/(f+1)})$.

Conversely, if there are $\Omega(n)$ nodes with non-zero outdegree and $p\in\omega(n^{-1/(f+1)})$, with probability $1-o(1)$ all copies of at least one such node $v$ are faulty. If $v$ sends a message under $A$, but all corresponding messages of copies of $v$ are not sent, the simulation fails. This shows that in this case the reinforcement is not valid. ∎

Efficiency of the Reinforcement

For $f\in{\mathbb{N}}$, we have that $\nu=\ell=f+1$ and $\eta=\ell^{2}=f^{2}+2f+1$, while we can sustain $p\in o(n^{-1/(f+1)})$. In the special case of $f=1$, we improve from $p\in o(1/n)$ for the original network to $p\in o(1/\sqrt{n})$ by doubling the number of nodes and quadrupling the number of edges.

VI More Efficient Reinforcement

In this section, we reduce the overhead in terms of edges at the expense of obtaining reinforcements that are not strong. We stress that the obtained trade-off between redundancy ($\nu$ and $\eta$) and the sustainable probability of faults $p$ is asymptotically optimal: as we require to preserve arbitrary routing schemes in a blackbox fashion, we need sufficient redundancy on the link level to directly simulate communication. From this observation, both for Om$\left(p\right)$ and Byz$\left(p\right)$ we can readily derive trivial lower bounds on redundancy that match the constructions below up to lower-order terms.

VI-A A Toy Example

Before we give the construction, we give some intuition on how we can reduce the number of required edges. Consider the following simple case. $G$ is a single path of $n$ vertices $(v_{1},\ldots,v_{n})$, and the schedule requires that in round $i$, a message is sent from $v_{i}$ to $v_{i+1}$. We would like to use a “budget” of only $n$ additional vertices and an additional $(1+\varepsilon)m=(1+\varepsilon)(n-1)$ links, assuming the fault model Om$\left(p\right)$. One approach is to duplicate the path and extend the routing scheme accordingly. We already used our entire budget apart from $\varepsilon m$ links! This reinforcement is valid as long as one of the paths succeeds in delivering the message all the way. The probability that one of the paths “survives” is $1-(1-(1-p)^{n})^{2}\leq 1-(1-e^{-pn})^{2}\leq e^{-2pn}$, where we used that $1-x\leq e^{-x}$ for any $x\in\mathbb{R}$. Hence, for any $p=\omega(1/n)$, the survival probability is $o(1)$. In contrast, the strong reinforcement with $\ell=2$ (i.e., $f=1$) given in § V sustains any $p\in o(1/\sqrt{n})$ with probability $1-o(1)$; however, while it adds $n$ nodes only, it requires $3m$ additional edges.

We need to add some additional edges to avoid that the likelihood of the message reaching its destination drops too quickly. To this end, we use the remaining $\varepsilon m$ edges to “cross” between the two paths every $h\triangleq 2/\varepsilon$ hops (assume $h$ is an integer), cf. Figure 1. This splits the path into segments of $h$ nodes each. As long as, for each such segment, in one of its copies all nodes survive, the message is delivered. For a given segment, this occurs with probability $1-(1-(1-p)^{h})^{2}\geq 1-(ph)^{2}$. Overall, the message is thus delivered with probability at least $(1-(ph)^{2})^{n/h}\geq 1-nhp^{2}$. As for any constant $\varepsilon$, $h$ is a constant, this means that the message is delivered a.a.s. granted that $p\in o(1/\sqrt{n})$!

Remark 2**.**

The reader is cautioned to not conclude from this example that random sampling of edges will be sufficient for our purposes in more involved graphs. Since we want to handle arbitrary routing schemes, we have no control over the number of utilized routing paths. As the latter is exponential in $n$, the probability that a fixed path is not “broken” by $F$ would have to be exponentially small in $n$. Moreover, trying to leverage Lovász Local Lemma for a deterministic result runs into the problem that there is no (reasonable) bound on the number of routing paths that pass through a single node, i.e., the relevant random variables (i.e., whether a path “survives”) exhibit lots of dependencies.

VI-B Partitioning the Graph

To apply the above strategy to other graphs, we must take into account that there can be multiple intertwined routing paths. However, the key point in the above example was not that we had path segments, but rather that we partitioned the nodes into constant-size regions and added few edges inside these regions, while fully connecting the copies of nodes at the boundary of the regions.

In general, it is not possible to partition the nodes into constant-sized subsets such that only a very small fraction of the edges connects different subsets; any graph with good expansion is a counter-example. Fortunately, many network topologies used in practice are good candidates for our approach. In the following, we will discuss grid networks and minor free graphs, and show how to apply the above strategy in each of these families of graphs.

Grid Networks

We can generalize the above strategy to hypercubes of dimension $d>1$.

Definition 4** (Hypercube Networks).**

A $q$-ary $d$-dimensional hypercube has node set $[q]^{d}$ and two nodes are adjacent if they agree on all but one index $i\in[d]$, for which $|v_{i}-w_{i}|=1$.

Lemma 3**.**

For any $h,d\in{\mathbb{N}}$, assume that $h$ divides $q\in{\mathbb{N}}$ and set $\varepsilon=1/h$. Then the $q$-ary $d$-dimensional hypercube can be partitioned into $(q/h)^{d}$ regions of $h^{d}$ nodes such that at most an $\varepsilon$-fraction of the edges connects nodes from different regions.

Proof.

We subdivide the node set into $h$-ary $d$-dimensional subcubes; for an example of the subdivision of the node set of a $6$-ary $2$-dimensional hypercube into $2$-ary $2$-dimensional subcubes see Figure 1. There are $(q/h)^{d}$ such subcubes. The edges crossing the regions are those connecting the faces of adjacent subcubes. For each subcube, we attribute for each dimension one face to each subcube (the opposite face being accounted for by the adjacent subcube in that direction). Thus, we have at most $dh^{d-1}$ crossing edges per subcube. The total number of edges per subcube are these crossing edges plus the $d(h-1)h^{d-1}$ edges within the subcube. Overall, the fraction of crossedges is thus at most $1/(1+(h-1))=1/h$, as claimed. ∎

Note that the above result and proof extend to tori, which also include the “wrap-around” edges connecting the first and last nodes in any given dimension.

Minor free Graphs

Another general class of graphs that can be partitioned in a similar fashion are minor free bounded-degree graphs.

Definition 5** ($H$-Minor free Graphs).**

For a fixed graph $H$, $H$ is a minor of $G$ if $H$ is isomorphic to a graph that can be obtained by zero or more edge contractions on a subgraph of $G$. We say that a graph G is $H$-minor free if $H$ is not a minor of $G$.

For any such graph, we can apply a corollary from [42, Coro. 2], which is based on [3], to construct a suitable partition.

Theorem 3** ([42]).**

Let $H$ be a fixed graph. There is a constant $c(H)>1$ such that for every $\varepsilon\in(0,1]$ and every $H$-minor free graph $G=(V,E)$ with degree bounded by $\Delta$ a partition $R_{1},\ldots,R_{k}\subseteq V$ with the following properties can be found in time $O(|V|^{3/2})$:

1. (i)

$\forall i:|R_{i}|\leq\frac{c(H)\Delta^{2}}{\varepsilon^{2}}$, 2. (ii)

$\forall i$* the subgraph induced by $R_{i}$ in $G$ is connected.* 3. (iii)

$|\{(u,v)\mid u\in R_{i},v\in R_{j},i\neq j\}|\leq\varepsilon\cdot|V|$.

Remark 3**.**

Grids and tori of dimension $d>2$ are not minor-free.

We note that this construction is not satisfactory, as it involves large constants. It demonstrates that a large class of graphs is amenable to the suggested approach, but it is advisable to search for optimized constructions for more specialized graph families before applying the scheme.

VI-C Reinforcement

Equipped with a suitable partition of the original graph $G=(V,E)$ into disjoint regions $R_{1},\ldots,R_{k}\subseteq V$, we reinforce as follows. As before, we set $V^{\prime}\triangleq V\times[\ell]$, denote $v_{i}\triangleq(v,i)$, define $P(v_{i})\triangleq v$, and set $\ell\triangleq f+1$. However, the edge set of $G^{\prime}$ differs. For $e=(v,w)\in E$,

[TABLE]

and we set $E^{\prime}\triangleq\bigcup_{e\in E}E_{e}^{\prime}$.

Simulation under Om$\left(p\right)$

Consider $v\in V$. We want to maintain the invariant that in each round, some $v_{i}$ has a copy of the state of $v$ in $A$. To this end, $v^{\prime}\in V^{\prime}$

1. (1)

initializes local copies of all state variables of $v$ as in $A$ and sets $\text{{know}}_{v^{\prime}}=\text{{true}}$; 2. (2)

sends on each link $(v^{\prime},w^{\prime})\in E^{\prime}$ in each round

• •

message $M$, if $P(v^{\prime})$ would send $M$ via $(P(v^{\prime}),P(w^{\prime}))$ when executing $A$ and $\text{{know}}_{v^{\prime}}=\text{{true}}$,

• •

a special symbol $\bot$ if $\text{{know}}_{v^{\prime}}=\text{{true}}$, but $v$ would not send a message via $(P(v^{\prime}),P(w^{\prime}))$ according to $A$, or

• •

no message if $\text{{know}}_{v^{\prime}}=\text{{false}}$; 3. (3)

if, in a given round, $\text{{know}}_{v^{\prime}}=\text{{true}}$ and $v^{\prime}$ receives for each neighbor $w$ of $P(v^{\prime})$ a message from some $w_{j}\in V^{\prime}$, it updates the local copy of the state of $v$ in $A$ as if $P(v^{\prime})$ received this message (interpreting $\bot$ as no message); and 4. (4)

if this is not the case, $v^{\prime}$ sets $\text{{know}}_{v^{\prime}}=\text{{false}}$.

We claim that as long as $\text{{know}}_{v^{\prime}}=\text{{true}}$ at $v^{\prime}$, $v^{\prime}$ has indeed a copy of the state of $P(v^{\prime})$ in the corresponding execution of $A$; therefore, it can send the right messages and update its state variables correctly.

Lemma 4**.**

Suppose that for each $k^{\prime}\in[k]$, there is some $i\in[\ell]$ so that $\{v_{i}\,|\,v\in R_{k^{\prime}}\}\cap F^{\prime}=\emptyset$. Then $A^{\prime}$ simulates $A$.

Proof.

Select for each $R_{k^{\prime}}$, $k^{\prime}\in[k]$, some $i$ such that $\{v_{i}\,|\,v\in R_{k^{\prime}}\}\cap F^{\prime}=\emptyset$ and denote by $C$ the union of all these nodes. As $P(C)=V$, it suffices to show that each $v^{\prime}\in C$ successfully maintains a copy of the state of $P(v^{\prime})$ under $A$. However, we also need to make sure that all messages, not only the ones sent by nodes in $c$, are “correct,” in the sense that a message sent over edge $(v^{\prime},w^{\prime})\in E^{\prime}$ in round $r$ would be sent by $A$ over $(P(v^{\prime}),P(w^{\prime}))$ (where $\bot$ means no message is sent). Therefore, we will argue that the set of nodes $T_{r}\triangleq\{v^{\prime}\in V^{\prime}\,|\,\text{{know}}_{v^{\prime}}=\text{{true}}\text{ in round }r\}$ knows the state of their counterpart $P(v^{\prime})$ under $A$ up to and including round $r\in{\mathbb{N}}$. As nodes $v^{\prime}$ with $\text{{know}}_{v^{\prime}}=\text{{false}}$ do not send any messages, this invariant guarantees that all sent messages are correct in the above sense.

We now show by induction on the round number $r\in{\mathbb{N}}$ that (i) each $v^{\prime}\in T_{r}$ knows the state of $P(v^{\prime})$ under $A$ and (ii) $C\subseteq T_{r}$. Due to initialization, this is correct initially, i.e., in “round [math];” we use this to anchor the induction at $r=0$, setting $T_{0}\triangleq V^{\prime}$.

For the step from $r$ to $r+1$, note that because all $v^{\prime}\in T_{r}$ have a copy of the state of $P(v^{\prime})$ at the end of round $r$ by the induction hypothesis, each of them can correctly determine the message $P(v^{\prime})$ would send over link $(v,w)\in E$ in round $r+1$ and send it over each $(v^{\prime},w^{\prime})\in E^{\prime}$ with $P(w^{\prime})=w$. Recall that $v^{\prime}\in T_{r+1}$ if and only if $v^{\prime}\in T_{r}$ and for each $(w,P(v^{\prime}))\in E$ there is at least one $w^{\prime}\in V^{\prime}$ with $P(w^{\prime})=w$ from which $v^{\prime}$ receives a message. Since under Om$\left(p\right)$ nodes in $F^{\prime}$ may only omit sending messages, it follows that $v^{\prime}\in T_{r+1}$ correctly updates the state variables of $P(v^{\prime})$, just as $P(v^{\prime})$ would in round $r+1$ of $A$.

It remains to show that $C\subseteq T_{r+1}$. Consider $v_{i}\in C$ and $(w,v)\in E$. If $v,w\in R_{k^{\prime}}$ for some $k^{\prime}\in[k]$, then $w_{i}\in C$ by definition of $C$. Hence, by the induction hypothesis, $w_{i}\in T_{r}$, and $w_{i}$ will send the message $w$ would send in round $r+1$ of $A$ over $(w,v)\in E$ to $v_{i}$, using the edge $(w_{i},v_{i})\in E^{\prime}$. If this is not the case, then there is some $j\in[\ell]$ such that $w_{j}\in C$ and we have that $(w_{j},v_{i})\in E^{\prime}$. Again, $v_{i}$ will receive the message $w$ would send in round $r+1$ of $A$ from $w_{j}$. We conclude that $v_{i}$ receives at least one copy of the message from $w$ for each $(w,v)\in E$, implying that $v\in T_{r+1}$ as claimed. Thus, the induction step succeeds and the proof is complete. ∎

Resilience of the Reinforcement

We denote $R\triangleq\max_{k^{\prime}\in[k]}\{|R_{k^{\prime}}|\}$ and $r\triangleq\min_{k^{\prime}\in[k]}\{|R_{k^{\prime}}|\}$.

Theorem 4**.**

The above construction is a valid reinforcement for Om$\left(p\right)$ if $p\in o((n/r)^{-1/(f+1)}/R)$. Moreover, if $G$ contains $\Omega(n)$ nodes with non-zero outdegree and $R\in O(1)$, $p\in\omega(n^{-1/(f+1)})$ implies that the reinforcement is not valid.

Proof.

By Lemma 4, $A^{\prime}$ simulates $A$ if for each $k^{\prime}\in[k]$, there is some $i\in[\ell]$ so that $\{v_{i}\,|\,v\in R_{k^{\prime}}\}\cap F^{\prime}=\emptyset$. For fixed $k^{\prime}$ and $i\in[\ell]$,

[TABLE]

Accordingly, the probability that for a given $k^{\prime}$ the precondition of the lemma is violated is at most $(Rp)^{f+1}$. As $k\leq n/r$, taking a union bound over all $k^{\prime}$ yields that with probability at least $1-n/r\cdot(Rp)^{f+1}$, $A^{\prime}$ simulates $A$. Therefore, the reinforcement is valid if $p\in o((n/r)^{-1/(f+1)}/R)$.

Now assume that $r\leq R\in O(1)$ and also that $p\in\omega(n^{-1/(f+1)})\subseteq\omega((n/r)^{-1/(f+1)}/R)$. Thus, for each $v\in V$, all $v^{\prime}\in V^{\prime}$ with $P(v^{\prime})=v$ simultaneously end up in $F^{\prime}$ with probability $\omega(1/n)$. Therefore, if $\Omega(n)$ nodes have non-zero outdegree, with a probability in $1-(1-\omega(1/n))^{\Omega(n)}=1-o(1)$ for at least one such node $v$ all its copies end up in $F^{\prime}$. In this case, the simulation fails if $v$ sends a message under $A$, but all copies of $v^{\prime}$ suffer omission failures in the respective round. ∎

Efficiency of the Reinforcement

For $f\in{\mathbb{N}}$, we have that $\nu=\ell=f+1$ and $\eta=(1-\varepsilon)\ell+\varepsilon\ell^{2}=1+(1+\varepsilon)f+\varepsilon f^{2}$, while we can sustain $p\in o(n^{-1/(f+1)})$. In the special case of $f=1$ and $\varepsilon=1/5$, we improve from $p\in o(1/n)$ for the original network to $p\in o(1/\sqrt{n})$ by doubling the number of nodes and multiplying the number of edges by $2.4$.

Remark 4**.**

For hypercubes and tori, the asymptotic notation for $p$ does not hide huge constants. Lemma 3 shows that $h$ enters the threshold in Theorem 4 as $h^{-d+1/2}$. For the cases of $d=2$ and $d=3$, which are the most typical (for $d>3$ grids and tori suffer from large distortion when embedding them into $3$-dimensional space), the threshold on $p$ degrades by factors of $11.2$ and $55.9$, respectively.

VI-D *Simulation under *Byz$\left(p\right)$

The same strategy can be applied for the stronger fault model Byz$\left(p\right)$, if we switch back to having $\ell=2f+1$ copies and nodes accepting the majority message among all messages from copies of a neighbor in the original graph.

Consider node $v\in V$. We want to maintain the invariant that in each round, a majority among the nodes $v_{i}$, $i\in[\ell]$, has a copy of the state of $v$ in $A$. For $v^{\prime}\in V^{\prime}$ and $(w,P(v^{\prime}))\in E$, set $N_{v^{\prime}}(w)\triangleq\{w^{\prime}\in V^{\prime}\,|\,(w^{\prime},v^{\prime})\in E^{\prime}\}$. With this notation, $v^{\prime}$ behaves as follows.

1. (1)

It initializes local copies of all state variables of $v$ as in $A$. 2. (2)

It sends in each round on each link $(v^{\prime},w^{\prime})\in E^{\prime}$ the message $v$ would send on $(P(v^{\prime}),P(w^{\prime}))$ when executing $A$ (if $v^{\prime}$ cannot compute this correctly, it may send an arbitrary message). 3. (3)

It updates its state in round $r$ as if it received, for each $(w,P(v^{\prime}))\in E$, the message the majority of nodes in $N_{v^{\prime}}(w)$ sent.

Lemma 5**.**

Suppose for each $k^{\prime}\in[k]$, there are at least $f+1$ indices $i\in[\ell]$ so that $\{v_{i}\,|\,v\in R_{k^{\prime}}\}\cap F^{\prime}=\emptyset$. Then $A^{\prime}$ simulates $A$.

Proof.

Select for each $R_{k^{\prime}}$, $k^{\prime}\in[k]$, $f+1$ indices $i$ such that $\{v_{i}\,|\,v\in R_{k^{\prime}}\}\cap F^{\prime}=\emptyset$ and denote by $C$ the union of all these nodes. We claim that each $v^{\prime}\in C$ successfully maintains a copy of the state of $P(v^{\prime})$ under $A$. We show this by induction on the round number $r\in{\mathbb{N}}$, anchored at $r=0$ due to initialization.

For the step from $r$ to $r+1$, observe that because all $v^{\prime}\in C$ have a copy of the state of $P(v^{\prime})$ at the end of round $r$ by the induction hypothesis, each of them can correctly determine the message $P(v^{\prime})$ would send over link $(v,w)\in E$ in round $r+1$ and send it over each $(v^{\prime},w^{\prime})\in E$ with $P(w^{\prime})=w$. For each $v^{\prime}\in C$ and each $(w,P(v^{\prime}))$, we distinguish two cases. If $P(v^{\prime})$ and $w$ are in the same region, let $i$ be such that $v^{\prime}=v_{i}$. In this case, $N_{v^{\prime}}(w)=\{w_{i}\}$ and, by definition of $C$, $w_{i}\in C$. Thus, by the induction hypothesis, $w_{i}$ sends the correct message in round $r+1$ over the link $(w^{\prime},v^{\prime})$. On the other hand, if $P(v^{\prime})$ and $w$ are in different regions, $N_{v^{\prime}}(w)=\{w_{i}\,|\,i\in[\ell]\}$. By the definition of $C$ and the induction hypothesis, the majority of these nodes (i.e., at least $f+1$ of them) sends the correct message $w$ would send over $(w,P(v^{\prime}))$ in round $r+1$ when executing $A$. We conclude that $v^{\prime}$ correctly updates its state, completing the proof. ∎

Resilience of the Reinforcement

As before, denote $R\triangleq\max_{k^{\prime}\in[k]}\{|R_{k^{\prime}}|\}$ and $r\triangleq\min_{k^{\prime}\in[k]}\{|R_{k^{\prime}}|\}$.

Theorem 5**.**

The above construction is a valid reinforcement for the fault model Byz$\left(p\right)$ if $p\in o((n/r)^{-1/(f+1)}/R)$. Moreover, if $G$ contains $\Omega(n)$ nodes with non-zero outdegree, $p\in\omega(n^{-1/(f+1)})$ implies that the reinforcement is not valid.

Proof.

By Lemma 5, $A^{\prime}$ simulates $A$ if for each $k^{\prime}\in[k]$, there are at least $f+1$ indices $i\in[\ell]$ so that $\{v_{i}\,|\,v\in R_{k^{\prime}}\}\cap F^{\prime}=\emptyset$. For fixed $k^{\prime}$ and $i\in[\ell]$,

[TABLE]

Thus, analogous to the proof of Theorem 1, the probability that for a given $k^{\prime}$ the condition is violated is at most

[TABLE]

By a union bound over the at most $n/r$ regions, we conclude that the precondition $p\in o((n/r)^{-1/(f+1)}/R)$ guarantees that the simulation succeeds a.a.s.

For the second statement, observe that for each node $v\in V$ of non-zero outdegree,

[TABLE]

Thus, a.a.s. there is such a node $v$. Let $(v,w)\in E$ and assume that $A$ sends a message over $(v,w)$ in some round. If $v$ and $w$ are in the same region, the faulty nodes sending an incorrect message will result in a majority of the $2f+1=|\{w^{\prime}\in V^{\prime}\,|\,P(w^{\prime})=w\}|$ copies of $w$ attaining an incorrect state (of the simulation), i.e., the simulation fails. Similarly, if $w$ is in a different region than $v$, for each copy of $w$ the majority message received from $N_{w^{\prime}}(v)$ will be incorrect, resulting in an incorrect state. ∎

Remark 5**.**

Note that the probability bounds in Theorem 5 are essentially tight in case $R\in O(1)$. A more careful analysis establishes similar results for $r\in\Theta(R)\cap\omega(1)$, by considering w.l.o.g. the case that all regions are connected and analyzing the probability that within a region, there is some path so that for at least $f+1$ copies of the path in $G^{\prime}$, some node on the path is faulty. However, as again we consider the case $R\in O(1)$ to be the most interesting one, we refrain from generalizing the analysis.

Efficiency of the Reinforcement

For $f\in{\mathbb{N}}$, we have that $\nu=\ell=2f+1$ and $\eta=(1-\varepsilon)\ell+\varepsilon\ell^{2}=1+(2+2\varepsilon)f+4\varepsilon f^{2}$, while we can sustain $p\in o(n^{-1/(f+1)})$. In the special case of $f=1$ and $\varepsilon=1/5$, we improve from $p\in o(1/n)$ for the original network to $p\in o(1/\sqrt{n})$ by tripling the number of nodes and multiplying the number of edges by $4.2$.

VII Empirical Evaluation

We have shown that our approach from § VI works particularly well for graphs that admit a certain partitioning, such as sparse graphs (e.g., minor-free graphs) or low-dimensional hypercubes. To provide some empirical motivation for the relevance of these examples, we note that the topologies collected in the Rocketfuel [58] and Internet Topology Zoo [35] projects are all sparse: almost a third (namely 32%) of the topologies even belong to the family of cactus graphs, and roughly half of the graphs (49%) are outerplanar [52].

To complement our analytical results and study the reinforcement cost of our approach in realistic networks, we conducted simulations on the around 250 networks from the Internet Topology Zoo. In the following, we will report on our main insights. Due to space constraints, we focus on the case of omission faults; the results for Byzantine faults follow the same general trends.

Recall that we replace each node by $f+1$ of its copies, and each edge with endpoints in different regions of the partition with $(f+1)^{2}$ copies; every other edge is replaced by $f+1$ copies. Our goal is to do this partitioning such that it minimizes the edge overhead of the new network and maximizes the probability of the network’s resilience. The fault probability of the network for given $p$, $f$ and partitions with $l_{1},l_{2},...,l_{k}$ nodes is calculated as $1-\prod_{i=1}^{k}[1-(1-(1-p)^{l_{i}})^{f+1}]$.

In the following, as a case study, we fix a target network failure probability of at most $0.01$. That is, the reinforced network is guaranteed to operate correctly with a probability of $99\%$, and we aim to maximize the probability $p$ with which nodes independently fail subject to this constraint. For this fixed target resilience of the network, we determine the value of $p$ matching it using the above formula. We remark that the qualitative behavior for smaller probabilities of network failure is the same, where the more stringent requirement means that our scheme outperforms naive approaches for even smaller network sizes.

For the examined topologies, it turned out that no specialized tools were needed to find good partitionings. We considered a Spectral Graph Partitioning tool [30] and Metis [34], a partitioning algorithm from a python library. For small networks (less than 14 nodes), we further implemented a brute-force algorithm, which provides an optimal baseline.

Figure 2 shows the resulting edge overheads for the different partitioning algorithms as a function of $p$ and for $f=3$, at hand of a specific example. For reference, we added the value of $p$ for the original graph ($f=0$) to the plot, which has an overhead factor of $1$ (no redundancy).

As to be expected, for each algorithm and the fixed value of $f=3$, as the number of components in partitionings increases, the edge overhead and $p$ increase as well. The “Singleton partition” point for $f=3$ indicates the extreme case where the size of the components is equal to 1 and the approach becomes identical to strong reinforcement (see § V); hence, it has an edge overhead of $(f+1)^{2}=16$. The leftmost points of the $f=3$ curves correspond to the other extreme of “partitioning” the nodes into a single set, resulting in naive replication of the original graph, at an edge overhead of $f+1=4$.

We observed this general behavior for networks of all sizes under varying $f$, where the spectral partitioning consistently outperformed Metis, and both performed very close to the brute force algorithm on networks to which it was applicable. We concluded that the spectral partitioning algorithm is sufficient to obtain results that are close to optimal for the considered graphs, most of which have fewer than 100 nodes, with only a handful of examples with size between 100 and 200. Accordingly, in the following we confine the presentation to the results obtained using the spectral partitioning algorithm.

In Figure 3, we take a closer look on how the edge overhead depends on $f$, at hand of a network of 33 nodes. Note that the partitionings do not depend on $f$, causing the 10 curves to have similar shape. As $f$ increases, the node overhead, edge overhead, and $p$ for the reinforced networks increase. We can see that it is advisable to use larger values of $f$ only if the strong reinforcement approach for smaller $f$ cannot push $p$ to the desired value. We also see that $f=1$ is sufficient to drive $p$ up to more than $6\%$, improving by almost two orders of magnitude over the roughly $0.01/33\approx 0.03\%$ the unmodified network can tolerate with probability $99\%$. While increasing $f$ further does increase resilience, the relative gains are much smaller, suggesting that $f=1$ is the most interesting case.

Following up on this, in Figure 4 we plot $p$ for all existing networks in the Topology Zoo using the spectral graph partitioning algorithm and $f=1$. Specifically, for each network, we calculated the value of $p$ on a set of reinforced networks with different node and edge overheads. Naturally, with increasing network size, the value of $p$ that can be sustained at a given overhead becomes smaller. Note, however, that naive replication quickly loses ground as $n$ becomes larger. In particular, already for about 20 nodes, an edge overhead of 3 with our approach is better than adding two redundant copies of the original network, resulting in more nodes, but the same number of edges. Beyond roughly 50 nodes, our approach outperforms two independent copies of the network using fewer edges, i.e., an edge overhead of 2.5.

To show more clearly when our approach outperforms naive network replication, Figure 5 plots the relative gain in the probability $p$ of node failure that can be sustained compared to the original network.

This plot is similar to the previous one. The y-axis now represents $p$ divided by the value of $p$ for the original graph. We now see that naive replication provides an almost constant improvement across the board. This is due to the fact that under this simple scheme, the reinforcement fails as soon as in each copy of the graph at least one node fails, as it is possible that a routing path in the original graph involves all nodes corresponding to failed copies.

Denote by $p_{k}$ the probability of node failure that can be sustained with $99\%$ reliability when simply using $k$ copies of the original graph (in particular $p_{1}\approx 0.01/n$). For small $k$, the probability $(1-p_{k})^{n}$ that a single copy of the original graph is fault-free needs to be close to $1$. Hence, we can approximate $(1-p_{k})^{n}\approx 1-p_{k}n$. The probability that all copies contain a failing node is hence approximately $(p_{k}n)^{k}$. Thus, $p_{1}n\approx 0.01\approx(p_{k}n)^{k}$, yielding that

[TABLE]

In particular, we can expect ratios of roughly $10$ for $k=2$ and $21.5$ for $k=3$, respectively. The small discrepancy to the actual numbers is due to the approximation error, which would be smaller for higher target resilience.

As the plot clearly shows, our method achieves a relative improvement that increases with $n$, as predicted by Theorem 4. In conclusion, we see that our approach promises substantial improvements over the naive replication strategy, which is commonly employed in mission-critical networks (e.g., using dual planes as in RFC 7855 [53]).

VIII Discussion

In the previous sections, we have established that constant-factor redundancy can significantly increase reliability of the communication network in a blackbox fashion. Our constructions in § VI are close to optimal. Naturally, one might argue that the costs are still too high. However, apart from pointing out that the costs of using sufficiently reliable components may be even higher, we would like to raise a number of additional points in favor of the approach.

Node Redundancy

When building reliable large-scale systems, fault-tolerance needs to be considered on all system levels. Unless nodes are sufficiently reliable, node replication is mandatory, regardless of the communication network. In other words, the node redundancy required by our construction may not be an actual overhead to begin with. When taking this point of view, the salient question becomes whether the increase in links is acceptable. Here, the first observation is that any system employing node redundancy will need to handle the arising additional communication, incurring the respective burden on the communication network. Apart from still having to handle the additional traffic, however, the system designer now needs to make sure that the network is sufficiently reliable for the node redundancy to matter. Our simple schemes then provide a means to provide the necessary communication infrastructure without risking to introduce, e.g., a single point of failure during the design of the communication network; at the same time, the design process is simplified and modularized.

Dynamic Faults

Because of the introduced fault-tolerance, faulty components do not impede the system as a whole, so long as the simulation of the routing scheme can still be carried out. Hence, one may repair faulty nodes at runtime. If $T$ is the time for detecting and fixing a fault, we can discretize time in units of $T$ and denote by $p_{T}$ the (assumed to be independent) probability that a node is faulty in a given time slot, which can be bounded by twice the probability to fail within $T$ time. Then the failure probabilities we computed in our analysis directly translate to an upper bound on the expected fraction of time during which the system is not (fully) operational.

Adaptivity

The employed node- and link-level redundancy may be required for mission-critical applications only, or the system may run into capacity issues. In this case, we can exploit that the reinforced network has a very simple structure, making various adaptive strategies straightforward to implement.

1. (i)

One might use a subnetwork only, deactivating the remaining nodes and links, such that a reinforced network for smaller $f$ (or a copy of the original network, if $f=0$) remains. This saves energy. 2. (ii)

One might subdivide the network into several smaller reinforced networks, each of which can perform different tasks. 3. (iii)

One might leverage the redundant links to increase the overall bandwidth between (copies of) nodes, at the expense of reliability. 4. (iv)

The above operations can be applied locally; e.g., in a congested region of the network, the link redundancy could be used for additional bandwidth. Note that if only a small part of the network is congested, the overall system reliability will not deteriorate significantly.

Note that the above strategies can be refined and combined according to the profile of requirements of the system.

IX Related Work

Robust routing is an essential feature of dependable communication networks, and has been explored intensively in the literature already.

Resilient Routing on the Network Layer

Many existing resilient routing mechanisms on the network layer can be categorized according to whether they are supported in the control plane, e.g., [8, 14, 23, 24, 28, 47], or in the data plane, e.g., [13, 22, 38, 43, 59, 61], see also the recent survey [11]. These mechanisms are usually designed to cope with link failures. Resilient routing algorithms in the control plane typically rely on a global recomputation of paths (either centralized [60], distributed [23] or both [46]), or on techniques based on link reversal [24], and can hence re-establish policies relatively easily; however, they come at the price of a relatively high restoration time [23]. Resilient routing algorithms in the dataplane can react to failures significantly faster [18]; however, due to the local nature of the failover, it is challenging to maintain network policies or even a high degree of resilience [12]. In this line of literature, the network is usually given and the goal is to re-establish routing paths quickly, ideally as long as the underlying physical network is connected (known as perfect resilience [18, 21]). In contrast, in this paper we ask the question of how to enhance the network in order to tolerate failures, considering also failures beyond link failures. In particular, we argue that such a re-enforced network simplifies routing as it is not necessary to compute new paths. The resulting problems are very different in nature, also in terms of the required algorithmic techniques.

Local Faults

In this paper, we consider more general failure models than typically studied in the resilient routing literature above, as our model is essentially a local fault model. Byzantine faults were studied in [15, 51] in the context of broadcast and consensus problems. Unlike its global classical counterpart, the $f$-local Byzantine adversary can control at most $f$ neighbors of each vertex. This more restricted adversary gives rise to more scalable solutions, as the problems can be solved in networks of degree $O(f)$; without this restriction, degrees need to be proportional to the total number of faults in the network.

We also limit our adversary in its selection of Byzantine nodes, by requiring that the faulty nodes are chosen independently at random. As illustrated, e.g., by Lemma 1 and Theorem 1, there is a close connection between the two settings. Informally, we show that certain values of $p$ correspond, asymptotically almost surely (a.a.s), to an $f$-local Byzantine adversary. However, we diverge from the approach in [15, 51] in that we require a fully time-preserving simulation of a fault-free routing schedule, as opposed to solving the routing task in the reinforced network from scratch.

Fault-Tolerant Logical Network Structures

Our work is reminiscent of literature on the design fault-tolerant network structures. In this area (see [49] for a survey), the goal is to compute a sub-network that has a predefined property, e.g., containing minimum spanning tree. More specifically, the sub-network should sustain adversarial omission faults without losing the property. Hence, the sub-network is usually augmented (with edges) from the input network in comparison to its corresponding non-fault-tolerant counterpart. Naturally, an additional goal is to compute a small such sub-network. In contrast, we design a network that is reinforced (or augmented) by additional edges and nodes so that a given routing scheme can be simulated while facing randomized Byzantine faults. As we ask for being able to “reproduce” an arbitrary routing scheme (in the sense of a simulation relation), we cannot rely on a sub-network.

The literature also considered random fault models. In the network reliability problem, the goal is to compute the probability that the (connected) input network becomes disconnected under random independent edge failures. The reliability of a network is the probability that the network remains connected after this random process. Karger [33] gave a fully polynomial randomized approximation scheme for the network reliability problem. Chechik et. al [10] studied a variant of the task, in which the goal is to compute a sparse sub-network that approximates the reliability of the input network. We, on the other hand, construct a reinforced network that increases the reliability of the input network; note also that our requirements are much stricter than merely preserving connectivity.

Self-healing

In the context of self-healing routing (e.g., Castañeda et al. [9]), researchers have studied a model where an adversary removes nodes in an online fashion, one node in each time step (at most $n$ such steps). In turn, the distributed algorithm adds links and sends at most $O(\Delta)$ additional messages to overcome the inflicted omission fault. Ideally, the algorithm is “compact”: each node’s storage is limited to $o(n)$ bits. A nice property of the algorithm in [9] is that the degrees are increased by at most $3$. For our purposes, an issue is that the diameter is increased by a logarithmic factor of the maximum initial degree, and hence the same holds for the latency of the routing scheme. Instead, we design a network that is “oblivious” to faults in the sense that the network is “ready” for independent random faults up to a certain probability, without the need to reroute messages or any other reconfiguration. Moreover, our reinforcements tolerate Byzantine faults and work for arbitrary routing schemes. We remark that compact self-healing routing schemes also deal with the update time of the local data structures following the deletion of a node; no such update is required in our approach.

Robust Peer-to-Peer Systems

Peer-to-peer systems are often particularly dynamic and the development of robust algorithms hence crucial. Kuhn et. al [37] study faults in peer-to-peer systems in which an adversary adds and removes nodes from the network within a short period of time (this process is also called churn). In this setting, the goal is to maintain functionality of the network in spite of this adversarial process. Kuhn et al. [37] considered hypercube and pancake topologies, with a powerful adversary that cannot be “fooled” by randomness. However, it is limited to at most $O(\Delta)$ nodes, where $\Delta$ is the (maximum) node degree, which it can add or remove within any constant amount of time. The main idea in [37] is to maintain a balanced partition of the nodes, where each part plays the role of a supernode in the network topology. This is done by rebalancing the nodes after several adversarial acts, and increasing the dimensionality of the hypercube in case the parts become too big.

In § VI, we also study hypercube networks. We employ two partitioning techniques to make sure that: (1) the size of each part is constant and (2) the number of links in the cut between the parts is at most $\varepsilon\cdot n$, where $n$ is the number of nodes. These partitioning techniques help us to dial down the overheads within each part, and to avoid a failure of each part due to its small size. However, we note that our motivation for considering these topologies is that they are used as communication topologies, for which we can provide good reinforcements, rather than choosing them to exploit their structure for constructing efficient and/or reliable routing schemes (which is of course one, but not the only reason for them being used in practice).

X Conclusion

In this paper, we proposed simple replication strategies for improving network reliability. Despite being simple and general, both in terms of their application and analysis, our strategies can substantially reduce the required reliability on the component level to maintain network functionality. The presented transformations allow us to directly reuse non-fault-tolerant routing schemes as a blackbox, and hence avoid the need to refactor working solutions. Hence being prepared for non-benign faults can be simple, affordable, and practical, and therefore enables building larger reliable networks. Interestingly, while our basic schemes may hardly surprise, we are not aware of any work systematically exploring and analyzing this perspective.

We understand our work as a first step and believe that it opens several interesting avenues for future research. For example:

• (i)

Which network topologies allow for good partitions as utilized in § VI? Small constants here result in highly efficient reinforcement schemes, which are key to practical solutions.

• (i)

Is it possible to guarantee strong simulations at smaller overheads?

• (i)

Can constructions akin to the one given in § VI be applied to a larger class of graphs?

On the practical side, while our simulations indicate that our approach can be significantly more efficient than state-of-the-art approaches to provision dependable ISP networks, it will be interesting to extend these empirical studies and also consider practical aspects such as the incremental deployment in specific networks.

Acknowledgments. This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement 716562) and from the Vienna Science and Technology Fund (WWTF), under grant number ICT19-045 (project WHATIF).