FAVE: A fast and efficient network Flow AVailability Estimation method with bounded relative error

TL;DR

FAVE is a novel, fast, and efficient method for estimating network flow availability with bounded error, significantly reducing computational costs compared to traditional Monte Carlo and importance sampling techniques.

Contribution

The paper introduces FAVE, a new sequential importance sampling approach that achieves bounded or vanishing relative error with linear complexity for flow availability estimation.

Findings

Reduces estimation cost by up to 900 times compared to Monte Carlo.

Achieves bounded or vanishing relative error with linear complexity.

Improves capacity planning accuracy with better flow availability guarantees.

Abstract

This paper focuses on helping network providers to carry out network capacity planning and sales projection by answering the question: For a given topology and capacity, whether the network can serve current flow demands with high probabilities? We name such probability as "{\it flow availability}", and present the \underline{f}low \underline{av}ailability \underline{e}stimation (FAVE) problem, which is a generalisation of network connectivity or maximum flow reliability estimations. Realistic networks are often large and dynamic, so flow availabilities cannot be evaluated analytically and simulation is often used. However, naive Monte Carlo (MC) or importance sampling (IS) techniques take an excessive amount of time. To quickly estimate flow availabilities, we utilize the correlations among link and flow failures to figure out the importance of roles played by different links in flow failures, and design three "sequential importance sampling" (SIS) methods which achieve "bounded or even vanishing relative error" with linear computational complexities. When applying to a realistic network, our method reduces the flow availability estimation cost by 900 and 130 times compared with MC and baseline IS methods, respectively. Our method can also facilitate capacity planning by providing better flow availability guarantees, compared with traditional methods.

Figures

Tables

Table 1. Table 1: Accuracy, efficiency and computational cost (i.e., simulation steps) comparison

		Monte Carlo Method			Our method (SEED-VRE)
Flow	Theoretical Unavailability	Estimated Unavailability	Empirical Variance	Number of Simulations	Estimated Unavailability	Empirical Variance	Number of Simulations
1	$1.000999\times {10}^{-3}$	$1.300\times {10}^{-3}$	$1.29831\times {10}^{-7}$	10,000	$1.000999\times {10}^{-3}$	$6.35275\times {10}^{-23}$	10
2	$1.000000\times {10}^{-3}$	$8.000000\times {10}^{-4}$	$7.99360\times {10}^{-8}$	10,000	$1.000000\times {10}^{-3}$	$2.11758\times {10}^{-23}$	10

Table 2. Table 2: A comparison between classical network reliabilities and flow availability

Reliability Measurement	NCR [7, 8, 9, 17, 18]	MFR [11, 12, 13]	Flow Availability
Definition	$\mathbb{P}[$node $u$ and $v$ are connected$]$	$\mathbb{P}[$max flow from $u$ to $v\ge \zeta ]$	$\mathbb{P}[$flow $f_i$’s demand is satisfied$]$
Required Information	Topology Information: $G\left(V,E\right)$ and $𝒑$.​​	Topology Information: $G\left(V,E\right)$, $𝒑$ and $𝒄$.​​​​	Topology Information: $G\left(V,E\right)$, $𝒑$ and $𝒄$; Flow Information: $F$ and $(s_i,t_i,d_i)$ for $\forall f_i\in F$.​​​
Application scenarios	1) Topology design evaluation.	1) Topology design evaluation; 2) Capacity planning design evaluation.​​	1) Topology design evaluation; 2) Capacity planning design evaluation; 3) Sales projection design evaluation.​​
Relationship	The flow availability is a generalization of network connectivity and maximum flow based reliabilities. When link capacity $c_i=\infty$ for $\forall i$, $\mathbb{P}[$flow $f_i$’s demand is satisfied$]=\mathbb{P}[$node $u$ and $v$ are connected$]$. When flow set $F$ contains only one flow, $\mathbb{P}[$flow $f_i$’s demand is satisfied$]=\mathbb{P}[$max flow from $u$ to $v\ge \zeta ]$.
Note: Consider in network $G(V,E)$, there exist node $u,v$ and flow $f_i$ from $u$ to $v$ with demand $\zeta$. Let vector $𝒑$ and $𝒄$ denote information of failure probability and capacity across all links. Consider in flow set $F$, each flow $f_i$ is associated with a source $s_i$, a destination $t_i$ and a demand $d_i$.

Table 3. Table 3: Important Notations

Notations	Descriptions
$N_l,N_f$	The number of transportation links and flows.
$e_i$	The $i$th link, where $i\in \{1,\cdots ,N_l\}$.
$p_i$, $p_i(x_i)$, $𝒑$, $p\left(𝒙\right)$	$p_i$ and $p_i\left(x_i\right)=p_i^{x_i}{\left(1-p_i\right)}^{1-x_i}$ are $e_i$’s failure probability and distribution. $𝒑=(p_1,\dots ,p_{N_l})$ and $p\left(𝒙\right)$ is the distribution induced by $𝒑$.​​
$𝒙$	Failure configuration. $𝒙=(x_1,\dots ,x_{N_l})$ and $x_i=1$ ($x_i=0$) means that the link is down (up).​​
$f_i$	The $i$th flow, where $i\in \{1,\cdots ,N_f\}$.
$\left(s_i,t_i,d_i,o_i\right)$	$f_i$’s flow information, including source $s_i$, destination $t_i$, bandwidth demand $d_i$ and availability target $o_i$.
$ℛ$	Indicator function of traffic engineering simulation. Given $𝒙$, $ℛ\left(𝒙\right)=1$ ($ℛ\left(𝒙\right)=0$) if the flow fails (succeeds).
$L,ℒ$	$L\subseteq \{1,\dots ,N_l\}$ is a link set. $ℒ$ is a collection of link sets.​​
$\Psi \left(L\right)$,  ${\Psi }^{-1}\left(𝒙\right)$	$\Psi \left(L\right)$ maps a link set $L$ to the failure configuration $𝒙$ which satisfies: $\forall i\in L$, $x_i=1$; $\forall i\notin L$, $x_i=0$. ${\Psi }^{-1}\left(𝒙\right)$ maps a failure configuration $𝒙$ to the link set $L=\left\{i\right|x_i=1,i\in \{1,\cdots ,N_l\}\}$.​​
$\mathrm{span}\left(L\right)$, $\mathrm{span}\left(ℒ\right)$	$\mathrm{span}\left(L\right)$ is the collection of link set $L$’s supersets and $\mathrm{span}\left(ℒ\right)$ is the collection of supersets of all link sets $L\in ℒ$.
$ℱ$	$ℱ=\left\{L\right|ℛ\left(\Psi \left(L\right)\right)=1,L\subseteq \{1,\cdots ,N_l\}\}$, the collection of all failure link sets which can result in flow failures.
$S,𝒮$	A SEED $S$ is a special link set satisfying: 1) $S\in ℱ$; 2) $\forall L⫋S$, $L\notin ℱ$; 3) $\forall L\supseteq S$, $L\in ℱ$. And $𝒮$ is the collection of all SEEDs.

Table 4. Table 4: An example for the IS method

$𝒙$	$(1,1,0)$	$(1,1,1)$	$(0,1,1)$
$q^{\ast }\left(𝒙\right)=\mathbb{P}\left[𝒙|ℛ\left(𝒙\right)=1\right]$	0.49975	0.00050	0.49975
$q\left(𝒙\right)={\prod }_{i=1}^{N_l}\mathbb{P}\left[x_i|ℛ\left(𝒙\right)=1\right]$	0.25000	0.25025	0.25000

Table 5. Table 5: Flow unavailability analysis results for the network in Fig. 3

		Full SEED Set					Partial SEED Set
		MC	IS	SEED-ZV	SEED-BRE	SEED-VRE	SEED-ZV	SEED-BRE	SEED-VRE
$ϵ=0.05$	$\mu$	$3.808\times {10}^{-3}$	$3.808\times {10}^{-3}$	$3.808\times {10}^{-3}$	$3.808\times {10}^{-3}$	$3.808\times {10}^{-3}$	$3.520\times {10}^{-3}$	$3.520\times {10}^{-3}$	$3.520\times {10}^{-3}$
	${\sigma }_q^2$	$3.793\times {10}^{-3}$	$7.549\times {10}^{-5}$	$0$	$1.663\times {10}^{-6}$	$1.059\times {10}^{-8}$	$0$	$1.145\times {10}^{-7}$	$1.277\times {10}^{-9}$
	${\sigma }_q/\mu$	$1.617\times {10}^1$	$2.281$	$0$	$3.387\times {10}^{-1}$	$2.702\times {10}^{-2}$	$0$	$8.886\times {10}^{-2}$	$9.384\times {10}^{-3}$
$ϵ=0.01$	$\mu$	$1.419\times {10}^{-4}$	$1.419\times {10}^{-4}$	$1.419\times {10}^{-4}$	$1.419\times {10}^{-4}$	$1.419\times {10}^{-4}$	$1.393\times {10}^{-4}$	$1.393\times {10}^{-4}$	$1.393\times {10}^{-4}$
	${\sigma }_q^2$	$1.418\times {10}^{-4}$	$1.323\times {10}^{-7}$	$0$	$6.586\times {10}^{-10}$	$1.523\times {10}^{-13}$	$0$	$4.282\times {10}^{-11}$	$1.743\times {10}^{-14}$
	${\sigma }_q/\mu$	$8.395\times {10}^1$	$2.564$	$0$	$1.809\times {10}^{-1}$	$2.751\times {10}^{-3}$	$0$	$4.697\times {10}^{-2}$	$9.477\times {10}^{-4}$
$ϵ=0.001$	$\mu$	$1.392\times {10}^{-6}$	$1.392\times {10}^{-6}$	$1.392\times {10}^{-6}$	$1.392\times {10}^{-6}$	$1.392\times {10}^{-6}$	$1.389\times {10}^{-6}$	$1.389\times {10}^{-6}$	$1.389\times {10}^{-6}$
	${\sigma }_q^2$	$1.392\times {10}^{-6}$	$1.348\times {10}^{-11}$	$0$	$6.903\times {10}^{-15}$	$1.563\times {10}^{-20}$	$0$	$4.431\times {10}^{-16}$	$1.768\times {10}^{-21}$
	${\sigma }_q/\mu$	$8.48\times {10}^2$	$2.637$	$0$	$5.969\times {10}^{-2}$	$8.982\times {10}^{-5}$	$0$	$1.515\times {10}^{-2}$	$3.026\times {10}^{-5}$
Note: Information of flow $f_{i^{\ast }}$: 1) the tuple of source, destination and demand $(s_{i^{\ast }},t_{i^{\ast }},d_{i^{\ast }})=(4,3,20.25)$; 2) the full SEED set $𝒮=\{\{2,5\},\{7,8\},$ $\{2,3,8\},\{2,8,12\},\{5,6,7\},\{5,7,11\}\}$; 3) the partial SEED set with good coverage ${𝒮}^{\prime }=\{\{2,5\},\{7,8\}\}$.

Table 6. Table 6: Summary of simulation costs ( N 𝑁 N steps)

	MC	IS	SEED-BRE
Simulation cost ($N$)	$3.11\times {10}^{10}$	$7.78\times {10}^5$	$2.01\times {10}^5$

Equations

$$\mu ={\mathbb{E}}_p\left[\mathcal{R}(\mathbf{x})\right]=\int \mathcal{R}(\mathbf{x})p(\mathbf{x})\mathrm{d}\mathbf{x},$$

$${\hat{\mu }}_{\text{MC}}=\frac{1}{N}{\sum }_{k=1}^N\mathcal{R}({\mathbf{x}}^{(k)}),$$

$$\mu ={\mathbb{E}}_p\left[\mathcal{R}(\mathbf{x})\right]=\int \mathcal{R}(\mathbf{x})\frac{p(\mathbf{x})}{q(\mathbf{x})}q(\mathbf{x})\mathrm{d}\mathbf{x}={\mathbb{E}}_q\left[\mathcal{R}(\mathbf{x})\frac{p(\mathbf{x})}{q(\mathbf{x})}\right],$$

$${\hat{\mu }}_{\text{IS}}=\frac{1}{N}{\sum }_{k=1}^N\mathcal{R}({\mathbf{x}}^{(k)})\frac{p({\mathbf{x}}^{(k)})}{q({\mathbf{x}}^{(k)})}=\frac{1}{N}{\sum }_{k=1}^N\mathcal{R}({\mathbf{x}}^{(k)})\omega ({\mathbf{x}}^{(k)}).$$

$${\mathbb{V}}_p\left[{\hat{\mu }}_{\text{MC}}\right]=\frac{1}{N}{\mathbb{V}}_p\left[\mathcal{R}(\mathbf{x})\right]=\frac{1}{N}(\mu -{\mu }^2),$$

$${\mathbb{V}}_q\left[{\hat{\mu }}_{\text{IS}}\right]=\frac{1}{N}{\mathbb{V}}_q\left[\mathcal{R}(\mathbf{x})\omega (\mathbf{x})\right]=\frac{1}{N}\left({\mathbb{E}}_p\left[\mathcal{R}(\mathbf{x})\omega (\mathbf{x})\right]-{\mu }^2\right).$$

$$q^{\ast }(\mathbf{x})=\mathbb{P}\left[\mathbf{x}|\mathcal{R}(\mathbf{x})=1\right].$$

$$\mathcal{R}(\cdot ):(\mathbb{G},\mathbb{F},\mathbb{D},\mathbb{B})\to \{0,1\},$$

$$\mathcal{R}(\cdot ):\mathbf{x}\to \{0,1\}.$$

$$q(\mathbf{x})={\prod }_{i=1}^{N_l}{q}_i(x_i),$$

$$q_i(x_i)=\mathbb{P}\left[x_i|\mathcal{R}(\mathbf{x})=1\right].$$

$$\frac{{\mu }^2\text{KL}(q^{\ast }\Vert q)}{N}\le {\mathbb{V}}_q\left[{\hat{\mu }}_{\text{IS}}\right]\le \frac{{\mu }^2\sqrt{2\log 2\text{KL}(q^{\ast }\Vert q)}}{N{\min }_{\mathbf{x}}q(\mathbf{x})}.$$

$$q^{\ast }(\mathbf{x})={\prod }_{i=1}^{N_l}{q}_i(\mathbf{x})\mathrm{and}{q}_i(\mathbf{x})=\frac{\mathbb{P}(\mathcal{R}=1|{\mathbf{x}}_{1:i})}{\mathbb{P}(\mathcal{R}=1|{\mathbf{x}}_{1:i-1})}{p}_i(x_i).$$

$$\hat{\mathbb{P}}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]=O\left(\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]\right),$$

$$\hat{\mathbb{P}}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]=\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]+o\left(\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]\right).$$

$$\hat{\mathbb{P}}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]=\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]\left({\theta }_i+o(1)\right),$$

$$\frac{\hat{\mathbb{P}}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]}{\hat{\mathbb{P}}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i-1}\right]}=\frac{\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]}{\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i-1}\right]}\cdot \left(\frac{{\theta }_i}{{\theta }_{i-1}}+o(1)\right).$$

$$q(\mathbf{x})$$

$$={\prod }_{i=1}^{N_l}\frac{\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]}{\mathbb{P}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i-1}\right]}p(x_i)\cdot {\prod }_{i=1}^{N_l}\left(\frac{{\theta }_i}{{\theta }_{i-1}}+o(1)\right)$$

$$=q^{\ast }(\mathbf{x})\cdot \left({\prod }_{i=1}^{N_l}\frac{{\theta }_i}{{\theta }_{i-1}}+o(1)\right).$$

$${\sigma }_q^2={\mu }^2\int \frac{q^{\ast }(\mathbf{x}{)}^2}{q(\mathbf{x})}\mathrm{d}\mathbf{x}-{\mu }^2={\mu }^2\left({\prod }_{i=1}^{N_l}\frac{{\theta }_{i-1}}{{\theta }_i}-1+o(1)\right)=O({\mu }^2).$$

$$\Psi (\cdot ):L\to \mathbf{x},\mathrm{where}\forall i\in L,x_i=1;\forall i\notin L,x_i=0.$$

$${\Psi }^{-1}(\cdot ):\mathbf{x}\to L,\mathrm{where}L=\{i|x_i=1\}.$$

$$\mathcal{F}\triangleq \left\{L|\mathcal{R}(\Psi (L))=1,L\subseteq \Omega \right\}.$$

$$\mathrm{span}\left(L\right)\triangleq \left\{L^{\prime }|L\subseteq L^{\prime }\subseteq \Omega \right\}.$$

$$\mathrm{span}\left(\mathcal{L}\right)\triangleq {\bigcup }_{L\in \mathcal{L}}\mathrm{span}\left(L\right).$$

$$\Phi (L)\triangleq {\prod }_{i\in L}{p}_i={\sum }_{L^{\prime }\in \mathrm{span}(L)}\mathbb{P}\left[\Psi \left(L^{\prime }\right)\right].$$

$$(1)S\in \mathcal{F};(2)\forall L⫋S,L\notin \mathcal{F};(3)\forall L\supseteq S,L\in \mathcal{F}.$$

$$(1)S({\mathbf{x}}_{1:i})\subseteq \{i+1,\cdots ,N_l\};$$

$$(2)S({\mathbf{x}}_{1:i})\cup {\Psi }^{-1}({\mathbf{x}}_{1:i})\in \mathcal{F};$$

$$\hat{\mathbb{P}}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]\leftarrow \hat{\mathbb{P}}\left[\mathcal{R}=1|{\mathbf{x}}_{1:i}\right]+(-1{)}^{|A|-1}\Phi \left({\bigcup }_{S({\mathbf{x}}_{1:i})\in A}S({\mathbf{x}}_{1:i})\right);$$

$$q(\mathbf{x})=\mathcal{M}(q^{(1)}(\mathbf{x}),\dots ,q^{(N_f)}(\mathbf{x})).$$

Full text

estimate flow availabilities, we utilize the correlations among link and flow failures to figure out the importance of roles played by different links in flow failures, and design three “sequential imp-ortance sampling” (SIS) methods which achieve “bounded or even vanishing relative error” with linear computational complexities.

When applying to a realistic network, our method reduces the flow​

availability estimation cost by 900 and 130 times compared with MC and baseline IS methods, respectively. Our method can also facilitate capacity planning by providing better flow availability guarantees, compared with traditional methods.

I Introduction

Network capacity planning is the process of ensuring sufficient bandwidth is provisioned so that service-level agreement (SLA) objectives like delay, jitter, loss and routing availability can be satisfied [1]. For the purpose of providing a better end user experience and at the same time, keeping the operation cost at an affordable level, effective capacity planning tools are crucial to network providers. Various systems have been built around this problem, such as Cisco’s MATE [2], Facebook’s Prophet [3], and Google’s backbone capacity planning tool [4].

Motivations of fast flow availability estimations: The above SLA objectives are called the demand of a traffic flow, and the

satisfaction probability of demand is defined as the flow availability. A key concern of capacity planning is in analyzing the effect of network changes or the arising of new flows on the flow demand satisfaction [2]. To illustrate, consider an example that a network provider needs to serve two flows as shown in Fig.1. In this network, each link $i$ is associated with a failure probability, say $p{=}0.001$ and a capacity $c_{i}$; also, flow routing follows the “max-min fairness” and “shortest path” policies. Each flow has a bandwidth demand, and an availability target (i.e., the lower bound probability that its requested bandwidth needs to be satisfied). The network provider will perform:

1. Flow availability testing, i.e., whether flow availability targets can be achieved? E.g., flow 1’s availability target is achieved if it obtains 10 units of bandwidth with a probability no less than $0.9999$.
2. Capacity planning, e.g, to improve the network, should the provider add more links between node $A$ and $B$, or add a new node $D$ so to increase the path diversity?
3. **Sales

projection**, e.g., to increase profit, can the provider admit a new​​

flow 3? Will the network still provide flow availability guarantees if flow 1 requires 20 additional units of bandwidth? All these cases need flow availabilities to see whether the network

(proposed by capacity planning) can serve flow demands (proposed by sales projection). We name the flow availability esti-mation problem as “FAVE”, and give a formal definition later.

For a large realistic network with complex failure patterns,

the flow availability often cannot be evaluated analytically, especially when the traffic scheduling is also considered. Hence, simulation (or sampling) is often used. The Monte Carlo (MC)

method [5], which simulates link failures with real link failure

  probabilities, is the most widely used.

Yet, it can be costly for MC to achieve a desired accuracy level, e.g., to estimate flow 2’s availability in Fig.1, by variances collected in Table​ 1, MC needs at least 3,840 simulation steps to guarantee the 95% confidence interval (CI) width below $10^{-3}$. To simulate a large network with many flows, it is crucial to find ways to speed up the simulation process.

High level ideas of our work: The flow availability estimation

can be more efficient if flow failures (i.e., flow demands could not be satisfied) happen more frequently by taking a proper distribution to simulate link failures, which is the key idea of “importance sampling (IS)” [6]. We first introduce a baseline IS solution for FAVE using “*the correlation between link failures

and flow failures*” to decide “important links” whose failures are more likely to result in flow failures. We then simulate such links’ failures more often to speed up simulating flow failures. To further improve upon the baseline IS, we show that some link sets’ failures happen more often, and play a role of “root causes” of flow failures. We name such link set as “SEED”. We

utilize “the correlation among link failures” captured by SEEDs​​

to decide “important link sets” whose failures are more likely

to result in flow failures. We propose three advanced “sequential importance sampling (SIS)” algorithms, among which the SEED-VRE algorithm has the “vanishing relative error (VRE)” and “linear computational complexity”. Table​ 1 illustrates the computational reduction of our methods: Using only 10 simulation steps, both empirical variance and estimation error are much smaller than that of MC with 10,000 simulation steps. In other words, not only we can speed up simulations, but the network operator is able to achieve a higher accuracy of flow availability estimation without incurring an additional cost.

We emphasize that flow availability estimation can also faci-litate capacity planning: for flows with unachievable availability targets, we show that allocating more capacities for the important links determined by our method would bring a better

flow availability improvement, compared with traditional me-thods to allocate more capacities for links with high utilizations.​​

Contributions: Our key contribution is in providing efficient and accurate solutions for FAVE. Contributions include:

• •

We formally define the network Flow AVailability Estimation​

(FAVE) problem, and show it generalizes the classical network reliability estimation problem [7, 8, 9, 10, 11, 12, 13].

• •

For FAVE with a single flow, we introduce a novel concept of

“SEED” and propose three advanced SIS algorithms which have attractive properties of bounded relative error (BRE) or

VRE with linear computational complexities.

• •

For FAVE with multiple flows, we propose a mixture SIS method which maintains BRE property and linear computa-tional complexity when estimating all flows’ availabilities.

• •

For FAVE with a partial SEED set information, we maintain the BRE and VRE properties of flow availability estimations.

• •

Extensive results show that our methods greatly speed up the flow availability estimation: 1) Given an illustrative network and full SEED sets or partial SEED sets with good coverage, SEED-VRE achieves variance reductions of 360,000 and 7,000 times in the single flow case, and *200,000 and 7 times

*for around 80% flows in the multiple flow case, compared with MC and baseline IS. 2) Given a realistic network and partial SEED sets with poor coverage, SEED-VRE reduces simulation cost by 900 and 130 times for around 80% flows in the multiple flow case, compared with MC and baseline IS.​

• •

We demonstrate our methods facilitate capacity planning by providing more accurate network reliability evaluations compared with classical methods, and greater flow availability improvements compared with solely using the link capacity utilization information.

Organizations: Section II introduces related work and some preliminaries. Section III formally defines the FAVE problem. In Section IV, we propose a baseline IS solution for FAVE and

show its error bounds, then we introduce “SEED” and present SEED based SIS solutions with better error bounds and linear

computational complexities. Section V considers more practi-cal issues, e.g., the multiple flows case and partial SEED sets information case. In Section VI, we evaluate our methods on both an illustrative network and a realistic network, i.e., the Abilene network [14]. Section VII shows the utility of our methods in capacity planning. Finally, Section VIII concludes.

II Related Work & Preliminaries

Our work can be viewed as a generalisation of previous work

in estimating network reliability [7, 8, 9, 10, 11, 12, 13] and is related with importance sampling [6]. Here, we briefly review previous rel-evant studies and compare them with our work. We also descr-ibe some concepts so readers can gain a better understanding of sampling methods for the network reliability estimation.

II-A Network Reliability Estimation

The most relevant literatures to our work focus on evaluating network reliability. To design reliable networks, one needs to measure the impact of network failures (e.g., link failures) on

network performances [15]. It is known that the exact computation of network reliability is #P-complete, and computational

complexities of all known algorithms are exponential increasing with the graph scale [16], which makes the problem intractable even for medium sized networks. Hence, most work on network reliability evaluation considers sampling methods to provide reliability estimations, and they can be classified into​

“network connectivity” based and “maximum flow” based.

Network connectivity reliability (NCR): NCR is a classical reliability measure adopted by most work [7, 8, 9]. The network​​

is modelled as a graph where links are either failed or operational, and NCR is measured by the probability that a given set

of nodes are connected when links fail with given probabilities. Authors in [7] take network repair policies into consideration to model link failures, and estimate NCR with the classical MC Authors in [8] combine MC with the particle swarm optimization to handle the NCR problem. To improve the efficiency of MC, authors in [9] apply the IS method and use pre-computed “graph minimal cuts” to approximate the optimal IS estimator.

Maximum flow reliability (MFR): Another line of work [11, 13, 12] generalizes the NCR problem by considering link capac-ities: link capacities are determined by link statuses, i.e., operational, failed or partially failed, which follow certain probability distributions. Given one source and one sink, MFR is defined as the probability that the maximum flow, i.e., the maximum achievable bandwidth from the source to the sink, is above a given threshold. In [11], link capacities are assumed to be continuous and the MC splitting method is applied for the

MFR estimation. Authors in [12] follow the idea of permutating MC and assume all links fail at the beginning and each one of them gets repaired after a random time. Authors in [13] consider estimating MFR with the order minimal cut sets.

Other reliabilities: Some works also study the connection availability [17] and service availability [18], and consider the probability that a connection or service is available. Authors in​

[17] evaluate the connection availability by computing the connection probability of a small subset of nodes exactly. Authors in [18] evaluate the service availability by using IS to estimate path availabilities. The problem considered in [17, 18] can be

transformed to a problem of determining the connectivity of certain nodes, where the network topology is given. Hence, [17, 18] are essentially the same with the NCR related work.

II-B Comparisons with Classical Reliability Estimation Work

We consider the “flow availability” as our reliability measure. We first give the definition of the flow demand.

Definition 1**.**

The “demand” of flow $f$ is the quality of service​​

(QoS) requirements decided by $f$’s SLA objective.

Consider different SLA objectives, the flow demand can be,

for instance, the bandwidth demand, latency demand or packet loss demand, which specifies $f$’s QoS requirement on bandwidth, transmission latency or packet loss.

To be concrete and so easier to understand, we take the bandwidth demand as an example, and the following analysis works the same for other demands. We define the flow availability as:

Definition 2**.**

For a given topology information, flow information, routing policy and resource allocation policy, the “flow availability” of $f$ is the probability that $f$’s demand is satisfied.​​​

Comparing with the state-of-the-art methods, our methods have the following advantages:

1. The flow availability can be applied to evaluate both NCR, given all links have unlimited capacities, and MFR, given the network contains only one flow. Yet, neither NCR nor MFR can address FAVE. To illustrate, consider the example in Fig.1. If link 1 fails, the network is still connected but neither flow 1 nor 2’s demands can be satisfied. Also, the maximum flow from $A$

to $B$ still achieves 10 units, but it does not imply flow 1 succeeds, which depends both on resource allocation and routing policies and other competing flows.

2. Flow availability can be applied to evaluate not only the reli-ability of network designs, including topology design and capacity planning, but also the feasibility of sales projection. In contrast, NCR only applies to the topology design evaluation and MFR only applies to the capacity planning evaluation, for they utilize solely the (partial) topology information.

To summarize, FAVE generalizes the NCR and MFR estimations and considers a more realistic problem setting. Moreover, it can be applied to evaluate impacts of more factors, e.g., network topology, capacity and flow information on network per-formances, and provides more accurate evaluation results. The detailed comparisons can be found in Table. 2.

II-C Sampling Methods for Network Reliability Estimations

Network reliability estimation problem: Let the network be

modelled as a directional multigraph $G{:=}({\kern-0.7ptV\!{,}E\kern-0.7pt})$ with $N_{v}$ nodes

in node set $V$ and $N_{l}$ links in link set $E$. Each link $e_{i}{\in}E$ is associated with a tuple $(p_{i},c_{i},x_{i})$ with a small probability $p_{i}$ to

represent $e_{i}$’s failure probability, a capacity $c_{i}$, and a status $x_{i}$,

where $x_{i}{=}1$ ($x_{i}{=}0$) means $e_{i}$ is down (up). Let $\boldsymbol{p}{=}\{p_{1},...,p_{N_{l}}\!\}$,​​

$\boldsymbol{c}{=}\{c_{1},...,c_{N_{l}}\!\}$ and $\boldsymbol{x}{=}\{x_{1},...,x_{N_{l}}\!\}$ be failure probability, capacity and status across all links respectively. $\boldsymbol{x}$, also called the “failure configuration”, is generated by the link failure distribution $p(\boldsymbol{x})$ induced by $\boldsymbol{p}$. There are $2^{N_{l}}$ possible configurations of $\boldsymbol{x}$, which is huge for a large realistic network.

Let $\mathcal{R}$ be the indicator function of some interested event $A$. According to the reliability definition, $A$ can refer to the event

that a subset of nodes are unconnected, or the maximum flow is

below the required threshold, or, in our case, the flow demand is unsatisfied. Given link statuses described by $\boldsymbol{x}$, $\mathcal{R}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}1$ if $A$ is observed and $\mathcal{R}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}0$ vice versa. The network unreliability, which is the occurrence probability of $A$, can be computed via the following integral in the discrete measure space:

[TABLE]

where $\mathbb{E}_{p}[\cdot]$ means taking the expectation over distribution $p({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. Then, the network reliability can be obtained by $1{-}\mu$.

Monte Carlo (MC) simulation: The MC simulation draws failure configurations $\boldsymbol{x}$ independently from $p({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ and estimate $\mu$ with the following MC estimator:

[TABLE]

where $N$ is the number of simulation steps and $\boldsymbol{x}^{({\kern-0.7ptk\kern-0.7pt})}$ is the $k$th generated failure configuration. As MC generates link statuses by true link failure probabilities (which can be small), it is rare

to observe link failures, and even rarer to observe $A$. This implies that we need a large $N$ to gain the desired accuracy.

Importance sampling (IS): To improve the efficiency of MC, IS changes the sampling distribution $p({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ to increase the occurrence of $A$, and assigns each sample $\boldsymbol{x}$ a weight to recover the unbiasedness. Specifically, it replaces Eq.(1) by:

[TABLE]

where $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ is the “importance distribution”. For convenience, denote $\omega({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}p({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})/q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ as the weight. Therefore, the above expectation is estimated by the following IS estimator:

[TABLE]

Estimator efficiency evaluation: An estimator’s efficiency is often measured by its “variance”. Take the MC estimator as an example, its variance is given by:

[TABLE]

where $\mathbb{V\!}_{p}[\cdot]$ means taking the variance over the distribution $p({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. The IS estimator’s variance can be expressed as:

[TABLE]

Note that the MC estimator is a special case of the IS estimator if $p({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$, $\forall\boldsymbol{x}$. Define $\sigma_{q}^{2}{=}\mathbb{V}_{q}\left[\mathcal{R}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})\omega({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})\right]$ as the “one-run variance” of the IS estimator. To achieve a desired estimation

accuracy, the CI width should be bounded by a threshold $\delta$, i.e.,

${2\alpha}{\sigma_{q}}{/\!}\sqrt{N}{\leq}{\delta}$, and the simulation cost is $N{\geq}({\kern-0.7pt2\alpha\sigma_{q}/\!\delta\kern-0.7pt})^{2}$, where $\alpha$ is a constant decided by the required confidence level. Thus, a small and bounded $\sigma_{q}^{2}$ implies an efficient estimator.

Zero-variance (ZV) importance distribution: The following theorem gives an optimal importance distribution $q^{*}\!({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$:

Theorem 1** (Zero-variance importance sampling).**

The IS estimator in Eq.(4) can achieve zero-variance, i.e, $\sigma_{q}^{2}{=}0$, if the importance distribution $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ where:

[TABLE]

Remark: Although the ZV property implies a minimum simulation cost, it is non-trivial and sometimes even impossible to compute the ZV importance distribution $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. Hence, the key of designing an efficient IS estimator lies in the approximation of $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. As for different applications and problem definitions,

the auxiliary information we can utilize and the way to approximate $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ are different, designing efficient IS estimators are highly challenging and problem-dependent.

III Problem Definition

Consider the network $G({\kern-0.7ptV,E\kern-0.7pt})$ with topology information $\boldsymbol{p},\newline \boldsymbol{c},\boldsymbol{x}$, as defined in Section II-C. We consider a flow set $F$ with $N_{f}{=}|F|$ flows and each flow $f_{i}{\in}F$ is associated with a tuple $(s_{i},t_{i},d_{i},o_{i})$ specifying $f_{i}$’s source $s_{i}$, destination $t_{i}$, demand $d_{i}$ and availability target $o_{i}$. We also define the following:

Definition 3**.**

A flow fails (succeeds) if its demand is unsatisfied (satisfied), e.g., the allocated bandwidth cannot (can) support its bandwidth demand.

We redefine the function $\mathcal{R}$ to indicate the network routing:

[TABLE]

where $\mathbb{G}$ represents the topology information, including a tuple $(p_{i},c_{i},x_{i})$ for every link $e_{i}{\in}E$; $\mathbb{F}$ represents the flow informa-tion, including a tuple $(s_{i},t_{i},d_{i},o_{i})$ for every flow $f_{i}{\in}F$; $\mathbb{D}$ and $\mathbb{B}$ represent the underlying routing and resource allocation policies, e.g., shortest path policy and max-min fairness policy. Function $\mathcal{R}$ outputs 1 if the interested flow fails and 0 other-wise. We assume all information in $({\kern-0.7pt\mathbb{G},\mathbb{F},\mathbb{D},\mathbb{B}\kern-0.7pt})$ is known, exc-ept link statuses described by $\boldsymbol{x}$. To simplify the expression, let:

[TABLE]

Namely, given all other information, the routing result indicated​

by $\mathcal{R}$ only depends on the failure configuration $\boldsymbol{x}$, which is generated by the link failure distribution $p({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ induced by $\boldsymbol{p}$.

The unavailability $\mu$, also called the flow failure probability, of a specific flow can be computed via the integral in Eq.(1). Our goal is to evaluate availabilities of (all) flows in $F$. Yet, the complexity of function $\mathcal{R}$ and the high dimensionality of the topological space make it impossible to evaluate the flow availability $1\!{-}\!\mu$ analytically. One alternative is to estimate $\mu$ via simulations. We name the network flow availability estimation problem as “FAVE”. And we can show it generalizes the classical network reliability estimation problem [7, 8, 9, 10, 11, 12, 13].

Theorem 2**.**

The FAVE problem generalizes both the network connectivity based and the maximum flow based network reliability estimation problems.

Remark: Due to the page limit, we only provide the sketch proof of Thm. 6 in this paper and we leave proofs of all other lemmas and theorems in the technical report [19].

In addition to measuring the estimation efficiency with the variance, we also consider two attractive error bound properties:​​

Definition 4** (Bounded Relative Error).**

An estimator with expectation $\mu$ and variance $\sigma^{2}/N$ has the bounded relative error (BRE) property

if $\sigma{=}O({\kern-0.7pt\mu\kern-0.7pt})$, i.e., the coefficient of variation (CV) $\varepsilon_{\emph{CV}}{\triangleq}\sigma/\mu$ satisfies $\lim_{\mu{\rightarrow}0}\varepsilon_{\emph{CV}}{<}\infty$.

Definition 5** (Vanishing Relative Error).**

An estimator with expectation $\mu$ and variance $\sigma^{2}/N$ has the vanishing relative error (VRE) property

if $\sigma{=}o({\kern-0.7pt\mu\kern-0.7pt})$, i.e., $\lim_{\mu{\rightarrow}0}\varepsilon_{\emph{CV}}{=}0$.

Remark: Note that the VRE property is stronger than the BRE property. The variance of MC is $\sigma^{2}{/}N{=}({\kern-0.7pt\mu{-}\mu^{2}\kern-0.7pt}){/}N$, i.e., $\sigma{=}O({\kern-0.7pt\!\sqrt{\mu}\kern-0.7pt})$. This implies that the MC estimator satisfies neither BRE nor VRE property. In the following, we will discuss how to design estimation methods which have above properties.

IV Algorithm Design

We first describe our design for “the single flow case”. We start with a baseline IS design to gain insights for efficient estimations. Then we introduce “SEED” and our SEED methods.

IV-A A Baseline Importance Sampling Design

1) ZV importance distribution approximation: It seems easy to design an IS estimator if we can well approximate $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ in Eq.(7). Yet, the following discussion shows that this is not an easy task. We use the KL divergence to measure the similarity between $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ and its approximation $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. We derive $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ by:

Theorem 3**.**

Assume the optimal importance distribution $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ in Eq.(7) is approximated by a product form distribution:

[TABLE]

then the KL divergence $\emph{KL}({\kern-0.7ptq^{*}\|q\kern-0.7pt})$ is minimized when:

[TABLE]

By now, the estimation of $\mathbb{P}\left[x_{i}|\mathcal{R}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}1\right]$ becomes a new problem. In fact, even given the exact $\mathbb{P}\!\left[x_{i}|\mathcal{R}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}1\right]$ expressions or

values, the performance of this IS method is still not guaran-teed: minimizing the KL divergence can only lower bound the

estimator’s variance, and the lower bound depends on how well​

$q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ in Eq.(11) can approximate $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. To see this, consider the case where a network has 2 nodes connected by 3 parallel links with $\boldsymbol{p}{=}({\kern-0.7pt0.001,0.2,0.001\kern-0.7pt})$. The flow fails if link statuses ${\bf\boldsymbol{x}}$ are $({\kern-0.7pt1{,}1{,}0\kern-0.7pt})$, $({\kern-0.7pt1{,}1{,}1\kern-0.7pt})$ or $({\kern-0.7pt0{,}1{,}1\kern-0.7pt})$. According to Thm. 3, one possible

importance distribution is $\boldsymbol{q}{=}({\kern-0.7pt0.50025,1,0.50025\kern-0.7pt})$. In this case, $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ is not well approximated by $q(\boldsymbol{x})$ in Table​ 4.

Remark: The above example illustrates that minimizing KL divergence cannot provide the IS method a performance guarantee: sometimes our chosen $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ can be very different from $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. A major reason is that the baseline IS assumes $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ has the product form in Eq.(10), i.e., it considers link failures are independent and ignores the correlations among them. We will discuss how to improve the design of $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ in later sections.

2) Estimation error bound analysis: The variance bounds of the baseline IS method is given by the following theorem:

Theorem 4**.**

If the IS estimator in Eq.(4) takes $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ in Eq.(11) as its importance distribution, then $\!\mathbb{V}_{\!q}{[}\hat{\mu}_{\emph{IS}}{]}\!$ is bounded by:

[TABLE]

Here $q^{*}\!$ is the optimal importance distribution given by Eq.(7).

Remark: Thm. 4 implies that although the variance of baseline IS is upper bounded, it can be worse than the variance of MC. This motivates us to seek for more efficient sampling methods with better error bounds.

IV-B Conditions for Efficient Sampling

In the baseline IS design, link importance distributions $q_{i}({\kern-0.7ptx_{i}\kern-0.7pt})$​

are assumed to be independent. This assumption simplifies the problem, but does not correspond to the reality. Consider the

example in Section IV-A, $\mathbb{P}[x_{3}|\mathcal{R}{=}1\!{,}x_{1}{=}1]$ greatly differs from​

$\mathbb{P}[x_{3}|\mathcal{R}{=}1{,}x_{1}{=}0]$, which implies the dependence among $q_{i}({\kern-0.7ptx_{i}\kern-0.7pt})$

and the correlation of links’ statuses cannot be ignored. Next, we propose our “sequential importance sampling” (SIS) based design and take this correlation into consideration.

1) ZV sequential importance sampling: Let us adapt the ZV importance distribution in Eq.(7) for the SIS method. Denote ${\bf\boldsymbol{x}}_{1:i}{=}({\kern-0.7ptx_{1},x_{2},{\cdots},x_{i}\kern-0.7pt})$ as the status of the first $i$th links.

Theorem 5**.**

For the FAVE problem, the IS estimator in Eq.(4) achieves the ZV property if the importance distribution $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){=}$ $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ for the SIS estimator, where:

[TABLE]

Remark: Different from baseline IS, SIS generates links’ statuses in a sequential manner, which enables its importance distribution $q_{i}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ to capture the correlation of links’ statuses, i.e., links’ importance distributions are dependent.

2) Conditions for good error bounds: To apply the above SIS estimator, we need to estimate (or approximate) $\mathbb{P}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$. The following theorem states that the above SIS is robust even if there exists some error when approximating $\mathbb{P}[\mathcal{R}{=}1|\boldsymbol{x}_{\!1{:}i}]$.

Theorem 6** (Conditions for BRE and VRE properties).**

The IS estimator in Eq.(4) has the BRE property if $\forall i{=}1,{\ldots},N_{l}$, $\hat{\mathbb{P}}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$ satisfies:

[TABLE]

and the VRE property if $\forall i{=}1,{\ldots},N_{l}$, $\hat{\mathbb{P}}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$ satisfies:

[TABLE]

Proof: Assume $\mathbb{P}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]{>}0$. Otherwise there is no need to generate the corresponding $\boldsymbol{x}$. Then, Eq.(14) is equivalent to:

[TABLE]

where $\theta_{i}$ is a constant. Thus, we can derive that:

[TABLE]

[TABLE]

[TABLE]

Namely, BRE property is achieved. As Eq.(15) is a special case

of Eq.(16) by restricting $\theta_{i}{=}1$ for all $i$, it is similar to show that

Eq.(15) implies $\sigma_{q}^{2}{=}o({\kern-0.7pt\mu^{2}\kern-0.7pt})$, i.e., VRE property is achieved.

Remark: Thm. 6 provides important guidelines to design sampling methods with BRE and VRE properties: the estimation $\hat{\mathbb{P}}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$ should satisfy the conditions listed in Thm. 6.

IV-C SEED Algorithms

1) SEED set and related definition: Before introducing how to​

approximate $\mathbb{P}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1{:}i}\right]$ for the SIS estimator, we first present​

some definitions used in the later discussion.

Let $\Omega{\triangleq}\left\{\!1,...,N_{l}\!\right\}$ be the index set of all links. We use function $\Psi({\kern-0.7pt\cdot\kern-0.7pt})$ and $\Psi^{{-}1}({\kern-0.7pt\cdot\kern-0.7pt})$ for transformations between the failure configuration $\boldsymbol{x}$ and the set of failure links $L{\subseteq}\Omega$:

[TABLE]

For one interested flow, denote the collection of all link sets whose failures can result in the flow failure by:

[TABLE]

Also, denote the collection of all supersets of link set $L$ by:

[TABLE]

Accordingly, for a collection of link sets $\mathcal{L}{=}\left\{L\right\}$,

[TABLE]

The probability of the event that “all links in $L$ fail” is:

[TABLE]

Definition 6** (SEED).**

Define the SEED $S$ as a special link set which satisfies the following conditions:

[TABLE]

We also denote the collection of all SEEDs by $\mathcal{S}$.

Definition 7** (Conditional SEED).**

Consider the statuses of some links are specified by $\boldsymbol{x}_{1{:}i}$, define the conditional SEED (cond-SEED for short) $S(\boldsymbol{x}_{1{:}i})$ as a special link set satisfying:

[TABLE]

Also, we denote the collection of all cond-SEEDs by $\mathcal{S}\left(\boldsymbol{x}_{1:i}\right)$.

Examples: We give some examples for the above definitions. Consider the example in Section I, we have $\mathcal{S}{=}\{\{1\}{,}\{4{,}5\}\!\}$. By the definition of SEED, the failure of any subset of a SEED, e.g., $L{=}\{4\}$ or $L{=}\{5\}$, will not result in flow 1’s failure; and the failure of any superset of a SEED, e.g., $L{=}\{1{,}2\}$, will result in flow 1’s failure. If given the first four links’ statuses by $\boldsymbol{x}_{1{:}4}{=}({\kern-0.7pt0,1,0,1\kern-0.7pt})$, there is only one cond-SEED $S({\kern-0.7pt\boldsymbol{x}_{1{:}4}\kern-0.7pt}){=}\{5\}$.

Next, we use “SEED” and “SEED set” to capture the correlation of link failures and approximate $\mathbb{P}\!\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$ in Thm.​ 5.

2) SEED based IS algorithms: Thm.​ 5 presents the optimal importance distribution $q^{*}\!({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ for the SIS method. We next show​

that $q^{*}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ can be computed exactly via Algorithm 1.

Theorem 7**.**

If failure configurations $\boldsymbol{x}$ are generated using SEED-ZV, the estimator in Eq.(4) has the ZV property.

Remark: Though SEED-ZV has the ZV property, it is compu-tationally expensive: as the need for traversing all combinations of cond-SEEDs $S({\kern-0.7pt\boldsymbol{x}_{1{:}i}\kern-0.7pt}){\in}\mathcal{S}$ for each link $e_{i}$, the computational complexity is $O({\kern-0.7ptN_{l}{\cdot}2^{\left|\mathcal{S}\right|}\kern-0.7pt})$.

Note that there is a tradeoff between estimation accuracy and computational complexity to estimate $q^{*}\!({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$. Next, we consider sacrificing some estimation accuracy of $q^{*}\!({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$, i.e., utilize probabilities of the most important cond-SEEDs for estimations (in line 3 of Algorithm 2), and propose SEED-BRE, which has a lower linear computational complexity. We show SEED-BRE has the BRE property by the following theorem.

Theorem 8**.**

If failure configurations $\boldsymbol{x}$ are generated using SEED-BRE, the estimator in Eq.(4) has the BRE property.

Remark: The computational complexity of SEED-BRE is only $O({\kern-0.7ptN_{l}\!\left|\mathcal{S}\right|\kern-0.7pt})$, as it needs to traverse all cond-SEEDs in $\mathcal{S}({\kern-0.7pt\boldsymbol{x}_{1:i}\kern-0.7pt})$, for each $e_{i}$. The size of $\mathcal{S}({\kern-0.7pt\boldsymbol{x}_{1{:}i\!}\kern-0.7pt})$ decreases when $i$ increases.

To further improve estimation accuracy, we utilize the probability sum of cond-SEEDs for estimations (in line 3 of Algo-rithm 3), and propose the SEED-VRE algorithm. We next show that SEED-VRE has the VRE property if link failures are rare.

Theorem 9**.**

When link failure probabilities are small and in the form of $p_{i}{=}O({\kern-0.7pt\epsilon\kern-0.7pt}),\forall i{\in}\Omega$. If failure configurations $\boldsymbol{x}$ are generated by SEED-VRE, the estimator in Eq.(4) has the VRE property.

Remark: The computational complexity of SEED-VRE is also $O({\kern-0.7ptN_{l}\!\left|\mathcal{S}\right|\kern-0.7pt})$, as it needs to traverse all the cond-SEED sets in $\mathcal{S}({\kern-0.7pt\boldsymbol{x}_{1:i}\kern-0.7pt})$, for each $e_{i}$.

By now, we have three SEED algorithms, i.e., SEED-ZV,

SEED-BRE and SEED-VRE, to compute the importance distributions of SIS estimator. They can achieve ZV, BRE and VRE properties respectively, and with the computational complexities $O({\kern-0.7ptN_{l}{\cdot}2^{\left|\mathcal{S}\right|}\kern-0.7pt})$, $O({\kern-0.7ptN_{l}\left|\mathcal{S}\right|\kern-0.7pt})$ and $O({\kern-0.7ptN_{l}\left|\mathcal{S}\right|\kern-0.7pt})$ respectively. However, all above discussions focus on the “single flow case”. Next, we will generalize our methods to handle multiple flows and take other practical issues into consideration.​​

V Practical Consideration

Previous discussions illustrate that SEED methods work well​

in estimating a single flow’s availability with the full informa-

tion of $\mathcal{S}$. Next, we consider more practical issues. First, as there are $O({\kern-0.7pt{N_{v}}^{\!\!2}\kern-0.7pt})$ flows in the network, it is costly to design a “customized” estimator for each flow and individually estimate

their availabilities. If the designed estimator works for a group of flows, the computational cost can be reduced significantly. Furthermore, as SEED methods rely on the SEED set which may be difficult to obtain the full information at times, we consider the case that only a partial information of $\mathcal{S}$ is available, e.g., we only know some frequently observed SEEDs.

V-A Generalization to Multiple Flows Case

To provide efficient and accurate availability estimations for a group of flows at the same time, one possibility is to utilize SEED methods to design a pure importance distribution $q^{({\kern-0.7ptk\kern-0.7pt})}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$​

for each flow $f_{k}{\in}F$, then take a mixture of these pure distributions with a strategy $\mathcal{M}$ to simulate link failures:

[TABLE]

To derive such a mixture importance distribution, we take a weighted sum of these pure distributions:

[TABLE]

Here $w_{k}$ can be viewed as the probability of taking $q^{({\kern-0.7ptk\kern-0.7pt})}\!({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ to

generate $\boldsymbol{x}$. Denote $\mu^{({\kern-0.7ptk\kern-0.7pt})}$ as $f_{k}$’s failure probability and $({\kern-0.7pt\sigma_{q}^{({\kern-0.7ptk\kern-0.7pt})}\kern-0.7pt})^{2}$

as the one-run variance when taking $q({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ to estimate $f_{k}$’s failure probability. Next we analyze error bounds of this mixture

sampling strategy, when applying to the multiple flows case.

Theorem 10**.**

Using the mixture sampling strategy in Eq.(27) with pure distributions $q^{({\kern-0.7ptk\kern-0.7pt})}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt})$ generated by SEED methods, the IS estimator achieves the BRE property for all flows availability estimations. Specifically, for flow $f_{k}$, the estimator’s one-run variance satisfies:

[TABLE]

Remark: Thm. 10 states that, when extending to the multiple flows case, our methods guarantee the estimation efficiency for all flows. Designing proper or even optimal weights $\!\{\!w_{k}\!\}$ is challenging. Online learning is a good approach to find a more efficient weight setting, and we leave this as a future work.

V-B Partial Seed Set Information

At times, it may be difficult to obtain a “full” SEED set $\mathcal{S}$, especially when the network is large and flow failures are rare. To provide robust estimations, consider the case that we have only a partial information of $\mathcal{S}$, e.g., limited historical data of

flow failures which gives $\mathcal{S}^{\prime}{\subset}\mathcal{S}$. Denote the cond-SEED set induced by $\mathcal{S}^{\prime}$ and $\boldsymbol{x}_{1:i}$ as $\mathcal{S}^{\prime}({\kern-0.7pt\boldsymbol{x}_{1{:}i}\kern-0.7pt})$. To analyze error bound properties of SEED algorithms, we provide the following lemma.

Lemma 1**.**

Given a partial SEED set $\mathcal{S}^{\prime}({\kern-0.7pt\boldsymbol{x}_{\!1{:}i}\kern-0.7pt})$, when estimating

$\mathbb{P}\!\left[\Psi^{{-}1}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){\in}\mathrm{span}\!\left(\mathcal{S}^{\prime}({\kern-0.7pt\boldsymbol{x}_{1{:}i}\kern-0.7pt})\right)\!|\boldsymbol{x}_{1{:}i}\right]$: SEED-ZV and SEED-BRE have ZV and BRE properties respectively; assume link failure probabilities are small and follow the form of $p_{i}{=}O({\kern-0.7pt\epsilon\kern-0.7pt}),\forall i{\in}\Omega$, SEED-VRE have the VRE property.

Remark: Thm.​ 5 states that the estimation accuracy depends on

how well $\mathbb{P}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$ is approximated. Given $\mathcal{S}$, SEED methods have good error bound properties for they can well approximate $\mathbb{P}\!\left[\Psi^{\!{-}1}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){\in}\mathrm{span}({\kern-0.7pt\mathcal{S}({\kern-0.7pt\boldsymbol{x}_{1{:}i}\kern-0.7pt})\kern-0.7pt})|\boldsymbol{x}_{1{:}i}\right]\!{=}\mathbb{P}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$. However,

given a partial SEED set $\mathcal{S}^{\prime}$, the bias between $\mathbb{P}\left[\mathcal{R}{=}1|\boldsymbol{x}_{1:i}\right]$ and $\mathbb{P}\left[\Psi^{{-}1}({\kern-0.7pt\boldsymbol{x}\kern-0.7pt}){\in}\mathrm{span}\!\left(\mathcal{S}^{\prime}({\kern-0.7pt\boldsymbol{x}_{1{:}i}\kern-0.7pt})\right)\!|\boldsymbol{x}_{1{:}i}\right]$ should be considered. Let us consider the following two cases:

Good coverage case. SEED methods maintain good error bound properties if $\mathcal{S}^{\prime}$ has a good coverage, which is defined formally as:

[TABLE]

Here is an example of such a partial SEED set:

[TABLE] 2. 2)

Poor coverage case. $\mathcal{S}^{\prime}$ may not have a good coverage, e.g.,

without prior knowledge of network and flow failures, $\mathcal{S}^{\prime}{=}\emptyset$. In such case, the SEED set information can be collected via pre-samplings and updated while simulating flow failures.

VI Evaluation of SEED methods

We evaluate our methods on both an illustrative small scale network and a realistic network with topology and traffic matrices extracted from the Abilene network [14]. The simulation cost to guarantee the estimation error (or relative error) below a constant $\delta$ is $N{\geq}\frac{4\alpha^{2}}{\delta^{2}}\sigma_{q}^{2}$ (or $N{\geq}\frac{4\alpha^{2}}{\delta^{2}}(\!\frac{\sigma_{q}}{\mu}\!)^{2}$). Hence, we use one-run variance $\sigma_{q}^{2}$ and coefficients of variation (CV) $\textstyle\varepsilon_{\emph{CV}}{=}\frac{\sigma_{q}}{\mu}$ to quantify the estimation efficiency. The variance reduction, i.e., $\textstyle\frac{\sigma_{\rm MC}^{2}}{\sigma_{\rm SEED}^{2}}$, can imply the simulation cost reduction, i.e., $\frac{N_{\rm MC}}{N_{\rm SEED}}$.

VI-A Experiments on an Illustrative Network

The illustrative network111This is provided so that readers can simulate and validate our methods. demonstrates the “best achievable theoretical improvements” using our method, compared with MC and baseline IS. We start with the single flow case, where full SEED sets or partial SEED sets with good coverage are provided. Then we extend it to the multiple flows case.

Experiment setting: The network is modelled as a directional multigraph $G$ with five nodes and 12 links as depicted in Fig.3.

For each link $e_{i}$: $p_{i}$ is uniformly distributed over $[0.5\epsilon,1.5\epsilon]$ ($\epsilon$ is a small positive number); $c_{i}$ is uniformly distributed over $\{50,80,100,200\}$. The flow set $F$ contains 18 flows. For each flow $f_{k}$: the source and destination are randomly selected; $d_{k}$ is the bandwidth demand and uniformly distributed over $[5,25]$. We consider traffic engineering follows the shortest path and max-min fairness policies. Note that the above setting provides an instance of network routing function $\mathcal{R}({\kern-0.7pt\cdot\kern-0.7pt})$ in Eq.(8).

Single flow analysis: We start with the single flow case, and

select one particular flow $f_{i^{*}}$ from the 18 flows. The detailed information of $f_{i^{*}}$, together with the SEED set information, are introduced in notes of Table​ 5. We compare our SEED methods with MC and baseline IS. The comparison result, including the expectation $\mu$, theoretical one-run variance $\sigma_{q}^{2}$ and CV $\varepsilon_{\emph{CV}}$, are summarized in Table​ 5. Let $\epsilon{=}0.05$: given a full SEED set $\mathcal{S}$, SEED-BRE and SEED-VRE achieve variance reductions of around 2,000 and 360,000 times compared with MC, and around 45 and 7,000 times compared with baseline IS; given a partial SEED set $\mathcal{S}^{\prime}$, our SEED methods estimate flow availabilities with very small biases and much smaller variances, i.e.,​​

with a small simulation cost, the estimation can be very close to the theoretical value. We also reduce $\epsilon$ from 0.05 to 0.001, to validate the vanishing property of SEED methods. While $\mu$

reduces with the decreasing $\epsilon$: $\varepsilon_{\emph{CV}}$ of MC increases significantly as we have discussed in Section III; $\varepsilon_{\emph{CV}}$ of baseline IS is relatively stable; $\varepsilon_{\emph{CV}}$ of SEED methods reduces significantly, and $\varepsilon_{\emph{CV}}$ of SEED-VRE even achieves a 300 times reduction.

Multiple flows analysis: Next, we take all 18 flows into consi-

deration. Let $\epsilon{=}0.001$. We consider an equally weighted sum of​

flows’ pure importance distributions as the mixture SIS distri-bution. Fig.3(a) illustrates cumulative distributions of $\varepsilon_{\emph{CV}}$ if pure distributions are generated by different methods. With SEED methods, for around 80% flows $\varepsilon_{\emph{CV}}{\leq}2$, which is smaller than the best case of $\varepsilon_{\emph{CV}}$ of baseline IS. This demonstrates the BRE property of SEED methods as stated in Thm. 10. Furthermore, both SEED methods and baseline IS, their $\varepsilon_{\emph{CV}}$ are 1,000 times smaller than that of MC. To depict the variance reduction

compared with MC much clearer, Fig.3(b) shows cumulative distributions of the variance reduction compared with MC. With SEED methods, more than 80% flows have variance reductions $\sigma_{\emph{MC}}^{2}/\sigma_{\emph{SEED}}^{2}{>}200,000$. To better illustrate the effici-ency improvement, Table​ 6 summarizes simulation costs to guarantee that for 80% flows, “with 95% confident the relative error is less than 0.01”, i.e., ${\alpha_{95}\sigma_{q}}/{\sqrt{N}}{\leq}0.01\hat{\mu}$.

VI-B Experiments on a Realistic Network

Next, consider a realistic network to demonstrate the “improvements in practice” by using our methods. As it is hard to

obtain the full SEED sets information in the complex realistic

case, simulations on the realistic network can validate the efficiency of SEED methods when estimating all flows’ availabilities, given partial SEED sets with poor coverage property.

Experiment setting: We use the Abilene network [14, 20] with topology and traffic matrices collected by [21]. The network contains 12 nodes and 30 links. The flow set contains 132 aggregated flows: all flows with the same source and destination are aggregated as a single flow222We take the Abilene network as an example and consider aggregated flows due to limitations of the accessible realistic traffic data.. We take each flow’s peak

(99 percentile) throughput [22] as its raw demand. As Abilene has a sufficient capacity to serve raw demands, we double raw demands to see whether the network can still support oversubscribed demands. The routing follows the shortest path policy. The capacity allocation follows the max-min fairness policy, which is also adopted by Google’s B4 backbone network [23].

Multiple flows analysis: We consider all aggregated flows and estimate their availabilities at the same time. Due to the high dimensionality of FAVE in this realistic network, it is costly to obtain theoretical variances of different methods. Thus, we run

each method 10,000 times and use the empirical variance $\hat{\sigma}_{q}^{2}$ [24] to estimate the one-run variance $\sigma_{q}^{2}$ and compute the simulation cost $N$ to guarantee that “with 95% confident the relative error is below 0.01”. Fig.5(a) shows cumulative distribu-tions of $N$ by taking the mixture of pure distributions gener-ated by different methods333Due to the exponential complexity, SEED-ZV is not applied in this case.. With SEED-VRE, we find that to achieve the desired accuracy level, for around 60% flows the required simulation costs $N{\leq}1,000$, and for around 80% flows $N{\leq}1,400$. Simulation costs for SEED-BRE, baseline IS and MC methods to guarantee 80% flows to achieve the accuracy target are 100,000, 180,000 and 1,260,000, respectively. So the

efficiency is improved by around 900 times via SEED-VRE and 13 times via SEED-BRE, compared with MC. Fig.5(b) illustr-ates cumulative distributions of the variance reduction compared with MC. With SEED methods, 80% of the flows have variance reductions larger than $900$ times.

VII Applications in Capacity Planning

Now, we demonstrate the utility of our method in capacity planning on the Abilene network.

Consider the case where a network provider needs to build new links with certain capacities so as to achieve all flow availability targets. For a capacity planning proposal $\mathcal{CP}$, we can evaluate its feasibility and adopt it if feasible, as follows:

• •

We obtain availability feedbacks of each flow $f_{i}$, including a flow availability estimation $\hat{\mu}_{i}$ with upper (lower) confidence bound $\textstyle\overline{\mu_{i}}$ ($\textstyle\underline{\mu_{i}}$) computed via empirical variances.

• •

$f_{i}$’s availability target $o_{i}$ is achieved if $\textstyle\underline{\mu_{i}}{\geq}o_{i}$ and unreached if $\textstyle\overline{\mu_{i}}{<}o_{i}$.​444We run enough simulations to guarantee $o_{i}{\not\in}[\underline{\mu_{i}},\overline{\mu_{i}}]$. $\mathcal{C\!P}$ is feasible if all flows’ availability targets are achieved; and infeasible otherwise.

By evaluating flow availabilities, we can not only determine which proposal allows the network to provide better flow availability guarantees, but also utilize flow availability feedbacks for further refinements of infeasible proposals. We consider the following information to refine an infeasible proposal:​​

• •

Link capacity utilization based: The utilization metric is the​​

primary metric of interest in capacity planning [1]. Hence, link capacity utilizations can imply the importance of links.

• •

Maximum flow based: The maximum flow value is a widely​

adopted network reliability measure [11, 12, 13]. We take the increase of the sum of maximum flows brought by increasing one unit link capacity to measure the importance of links.

• •

SEED based: By testing flow availabilities, we get $\mathcal{F}^{\prime}$, the set of flows with unsatisfied availability targets. The number

of failed flows in $\mathcal{F}^{\prime}$ when $e_{j}$ fails ($\sum_{f_{i}\!{\in}\mathcal{F}^{\prime}}\mathbb{P}\left[\mathcal{R}_{i}{=}1|x_{j}{=}1\right]$) can be estimated using SEED algorithms and can imply the importance of links.555As $\mathbb{P}[\mathcal{R}_{i}{=}1|\boldsymbol{x}_{1{:}k}]$, the probability that flow $f_{i}$ fails given the first $k$th links’ statues $\boldsymbol{x}_{\!1{:}k}$, can be estimated using SEED algorithms, $\mathbb{P}\left[\mathcal{R}_{i}{=}1|x_{j}{=}1\right]$ can also be estimated by taking $e_{j}$ as the first link and let $k{=}1$, $\boldsymbol{x}_{1:k}{=}1$.

Let the current capacity design of Abilene be the original proposal $\mathcal{CP}_{\!0}$, and twice the raw demands be flow demands, such that $\mathcal{CP}_{\!0}$ is infeasible. We rank links using the above metrics and summarize rankings in Fig.7. Assume the provider has a budget and only affords to build four new links, each has a capacity of 2.5Gbps and a failure probability of 0.01. Based on rankings in Fig.7, we have three proposals, i.e., $\mathcal{CP}_{\!1}$, $\mathcal{CP}_{\!2}$ and $\mathcal{CP}_{\!3}$ by taking the top four links in Fig.7, 7 and 7. Fig.7(a) shows flow availability evaluation results. As our method selects links with the largest impact on flow failures, it achieves greater flow availability improvements: in $\mathcal{CP}_{\!3}$, flow availabilities of around 80% of the flows reach 99.9%.

Insights 1: Improper capacity planning offers little help on improving flow availabilities. E.g., although $\mathcal{C\!P}_{\!2}$ maximizes the sum of maximum flows, it does not consider the distribution of traffic demands across the network, and thus only brings little improvements on flow availabilities.

Insights 2: Link utilization is not always the best indicator of capacity planning. With the shortest path routing policy, link $e_{i}$​

has a high capacity utilization if many flows’ shortest paths go through it. Yet, if it is easy to find some alternate link when $e_{i}$ fails, $e_{i}$’s failure will not result in flow failures and so $e_{i}$ is not

the most important if aiming at improving flow availabilities.

VIII Conclusion

In this paper, we propose fast and accurate methods in solving the FAVE problem. We introduce the concept of “SEED” to

determine the importance of roles played by different links in flow failures, and propose three SEED based SIS methods which achieve BRE and VRE properties with linear computational complexities. To provide robust and scalable estimations, we extend FAVE to the multiple flows case and partial SEED set case, and our methods maintain the estimation efficiency. We apply our methods on both an illustrative network and a realistic network, and our methods reduce the simulation cost by around 900 and 130 times compared with MC and baseline IS methods on the Abilene network. We show that our method facilitates capacity planning by providing more accurate network reliability estimations compared with classical methods, and greater flow availability improvements compared with solely using link capacity utilizations for capacity planning.