Practical Verifiable In-network Filtering for DDoS defense

TL;DR

This paper introduces VIF, a hardware-based trusted execution environment system that enables verifiable in-network filtering to improve DDoS attack mitigation and prevent misuse by transit networks.

Contribution

VIF provides a practical, scalable, and verifiable in-network filtering solution using commodity hardware TEEs, addressing trust issues in DDoS defense.

Findings

VIF handles 10 Gb/s traffic with 3,000 filter rules on commodity servers.

VIF scales to 500 Gb/s traffic and 150,000 rules through parallelization.

VIF can be deployed at IXPs for effective early adoption.

Abstract

In light of ever-increasing scale and sophistication of modern DDoS attacks, it is time to revisit in-network filtering or the idea of empowering DDoS victims to install in-network traffic filters in the upstream transit networks. Recent proposals show that filtering DDoS traffic at a handful of large transit networks can handle volumetric DDoS attacks effectively. However, the innetwork filtering primitive can also be misused. Transit networks can use the in-network filtering service as an excuse for any arbitrary packet drops made for their own benefit. For example, transit networks may intentionally execute filtering services poorly or unfairly to discriminate their competing neighbor ASes while claiming that they drop packets for the sake of DDoS defense. We argue that it is due to the lack of verifiable filtering - i.e., no one can check if a transit network executes the filter rules correctly as requested by the DDoS victims. To make in-network filtering a more robust defense primitive, in this paper, we propose a verifiable in-network filtering, called VIF, that exploits emerging hardware-based trusted execution environments (TEEs) and offers filtering verifiability to DDoS victims and neighbor ASes. Our proof of concept demonstrates that a VIF filter implementation on commodity servers with TEE support can handle traffic at line rate (e.g., 10 Gb/s) and execute up to 3,000 filter rules. We show that VIF can easily scale to handle larger traffic volume (e.g., 500 Gb/s) and more complex filtering operations (e.g., 150,000 filter rules) by parallelizing the TEE-based filters. As a practical deployment model, we suggest that Internet exchange points (IXPs) are the ideal candidates for the early adopters of our verifiable filters due to their central locations and flexible software-defined architecture.

Tables

Table 1. TABLE I : Execution times for the ILP solution and the greedy algorithm solution. The CPLEX’s mixed ILP solver is configured to stop when found sub-optimal solutions.

Number of rules ($k$)	CPLEX (sub-optimal)	Greedy
5,000	210.49s	0.31s
10,000	772.43s	0.50s
15,000	1,614.96s	0.73s

Table 2. TABLE II : Overhead of filter rule batch insertion to a multi-bit trie lookup table.

Number of rules in a batch	1	10	100	1000
Insert time (millisecond)	50	52	53	75

Table 3. TABLE III : Top five IXPs in each of the five regions. Numbers inside the parentheses denote the member sizes of the IXPs.

Rank	Europe	North America	South America	Asia Pacific	Africa
1	AMS-IX (1660)	Equinix Ashburn (598)	IX.br São Paulo (2082)	Equinix Singapore (504)	NAPAfrica Johannesburg (506)
2	DE-CIX (1494)	Any2 (557)	PTT Porto Alegre (258)	Equinix Sydney (393)	NAPAfrica Cape Town (258)
3	LINX Juniper (755)	SIX (462)	PTT Rio de Janeiro (246)	Megaport Sydney (383)	JINX (180)
4	EPIX Katowice (732)	TorIX (426)	CABASE-BUE (183)	BBIX Tokyo (286)	NAPAfrica Durban (122)
5	LINX LON1 (697)	Equinix Chicago (384)	PTT Curitiba (140)	HKIX (281)	IXPN Lagos (69)

Equations

$$\displaystyle\{\texttt{ALLOW},\texttt{DROP}\}\leftarrow f\big{(}\langle{p,a}\rangle,(\langle{p_{1},a_{1}}\rangle,\langle{p_{2},a_{2}}\rangle,\cdots)\big{)},$$

$$\displaystyle f\big{(}\langle{p,a}\rangle,(\langle{p_{1},a_{1}}\rangle,\langle{p_{2},a_{2}}\rangle,\cdots)\big{)}=f(p),$$

$$\text{Minimize}z$$

$$s.t.$$

$$\forall i:u\cdot \sum _{j=1}^n{y}_{i,j}+v\le M$$

$$\forall j:\sum _{i=1}^k{x}_{i,j}{y}_{i,j}\le G$$

$$\forall i:\sum _{j=1}^n{x}_{i,j}=b_i$$

$$\forall x_{i,j},y_{i,j}:(1-y_{i,j})x_{i,j}=0$$

$$\forall x_{i,j}\ge 0\text{and}\forall y_{i,j}\in \{0,1\}$$

Full text

I Introduction

Distributed denial-of-service (DDoS) attacks are highly prevalent. In the last decade, new attack strategies such as amplification [59] and new attack sources such as IoT devices [73] have surfaced, which have resulted in attacks of extremely high volume [71].

A large number of DDoS defenses have been extensively studied over the past two decades. Among them, an effective defense against the ever-increasing scale of DDoS attacks is in-network filtering or empowering DDoS victim networks to install in-network traffic filters in the upstream transit networks. This idea was proposed in early efforts (e.g., Pushback [47], D-WARD [49], AITF [9]) and has repeatedly re-surfaced in standardization committees (e.g., see the recent DDoS Open Threat Signaling [51]).

Dropping suspicious packets closer to the attack sources at the requests of DDoS victims is desirable because it (1) reduces wasted traffic on downstream ISPs, thereby reducing overall network bandwidth and cost of routing malicious traffic; and (2) has the potential to handle ever-increasing attack volume (e.g., several Tb/s) since attack volume at each distributed filtering point can be much lower than the aggregated volume at the victim [9].

Unsurprisingly, there is renewed interest in the community on revisiting in-network filtering solutions. Indeed, a recent DDoS defense architecture, called SENSS [58], suggests that the traffic filters installed at a few large transit ISPs directly by the remote DDoS victims can prevent most of the volumetric attack traffic from flooding the victim networks.

However, the in-network filtering primitive can be unfortunately misused by malicious transit networks. Transit networks may claim to provide in-network filtering service on behalf of a remote DDoS victim only to justify any arbitrary packet drops executed for their own benefit. For example, a transit network that offers in-network filtering service (which we call a filtering network) can easily compromise a filter rule [Drop 50% of HTTP flows destined to the victim network] requested by a DDoS victim to harm its neighbor networks, as illustrated in Figure 1. First, the malicious filtering network can silently discriminate its neighbor upstream ASes by applying arbitrarily modified filter rules (e.g., dropping 80% of HTTP traffic from AS $A$ but only 20% from AS $B$) based on its business preference (say, AS $A$ is AT&T and $B$ is Comcast). Discrimination and dispute among transit ISPs are not a new problem (e.g., a dispute between Level3 and Comcast [76]); however, it can be easily exacerbated if transit networks start dropping packets for the sake of DDoS defense. Worse yet, the filtering network can also perform the requested filter rules inaccurately (e.g., drop only 20% HTTP traffic on average) to save its filtering resources while claiming the faithful execution of the requested rules.

We argue that the verifiability of in-network filtering mitigates such misbehaviors by malicious filtering service providers. With filtering verifiability, when a filtering network executes a manipulated filter rule, the DDoS victim network who requested the filter rule or the direct neighbor autonomous system (ASes) of the filtering network can detect the misbehavior immediately.

In this paper, we offer the first technical means of the strong verifiability of filtering operations and develop a practical and scalable in-network filtering system, called VIF.111VIF stands for ‘Verifiable In-network Filtering’. VIF has a generic architecture designed for any transit networks (e.g., Tier-1, large Tier-2 ISPs, IXPs). We exploit software networking functions running on commodity hardware with trusted execution environments (TEEs), such as Intel SGX [20] and Arm TrustZone [10], which are widely available in commercial off-the-shelf (COTS) server platforms; see Microsoft and Google’s SGX-based cloud platforms [60, 56].

We identify two key technical challenges to realizing the VIF vision in practice:

• $\bullet$

Auditability. To make the in-network filtering verifiable, we exploit the trusted execution environment with hardware-based root of trust (e.g., Intel SGX) and provide integrity guarantee for filter logic. Yet, achieving auditable filtering operations is not trivial. One challenge is that the correct filtering operations are also affected by external inputs to the filter (e.g., incoming packet order, time clock feeds), which can be controlled by a malicious filtering network. Another challenge is that a malicious filtering network can still bypass the auditable filters by reconfiguring its network and avoid using the auditable filters for adversary-selected packets.

• $\bullet$

Scalability. To scale out the filter capacity for large volumetric attacks, parallelization with multiple TEE-based auditable filters is required; however, some necessary network components for parallelization (e.g., traffic load balancers) are not directly auditable. Moreover, distributing filter rules across multiple auditable filters creates an optimization challenge, which involves two dimensions of resource constraints.

Approach and Contributions.

VIF’s design makes the following key contributions to address these aforementioned challenges:

• $\bullet$

We analyze the requirements for auditable filters, particularly, their reliance on the external inputs to the filters, which can be controlled by malicious filtering networks (§III-A). Our key insight is that it is sufficient that traffic filters are stateless for auditable filter operations. We also implement an effective bypass detection that relies on the accountable packet logs (with an efficient sketch implementation) measured inside the TEE (§III-B). The packet logs can be used to identify packet drops/injections made outside of the auditable filters. We demonstrate that an efficient line-rate implementation (nearly 10 Gb/s throughput performance) of the auditable traffic filters with the TEE support is possible with several system optimizations (§V); and

• $\bullet$

For highly scalable filtering architecture, we implement a dynamic filter rule distribution algorithm across multiple auditable filters and untrusted network components [54, 26] (§IV). We implement a heuristic that can quickly reconfigure a large number of filter rules (e.g., 150,000 filter rules) and a large volume of incoming traffic (e.g., the total volume of 500 Gb/s) with auditable filter instances (e.g., 50 filters).

As an early deployment model, we suggest to deploy VIF in major Internet exchange points (IXPs) (§VI). In the last decade, IXPs have become the central infrastructure of the global Internet connectivity [7, 15], with large IXPs handling daily traffic volumes comparable to those carried by the largest Tier-1 ISPs, which makes the IXPs the perfect candidates for our verifiable filtering service [23]. We perform large-scale inter-domain simulations with two realistic attacks (i.e., DNS amplification, Mirai-based DDoS attacks) and show that deploying VIF in a small number (e.g., 5–25) of large IXPs is enough to handle the majority (e.g., up to 90%) of DDoS attacks (§VI-C).

II Problem Definition

In this section, we describe the threat model we consider in this paper (§II-A), the desired properties of the VIF design, the trusted execution environment model (§II-C) (§II-B) and the assumption we make in this work (§II-D).

II-A Threat Model

We consider a single malicious transit network that offers in-network filtering services for downstream networks. In this threat model, the adversary has full control of the control plane and data plane within its network.

Our threat model focuses on the problem of the malicious filtering network manipulating the DDoS-victim-submitted filter rules, which we call as filter rule violation attacks. In general, let us consider a filter rule $R$ requested by a victim network. The malicious filtering network may change $R$ arbitrarily into a different filter rule $R^{\prime}$ and apply it to all traffic destined to the victim network. Also, the malicious filtering network may apply different modified rules $R^{\prime}_{1},R^{\prime}_{2},\cdots$ for different traffic flows (e.g., packets delivered via different neighboring ASes).

In the followings, we present two example attack goals, where the manipulation of filter rules at a filtering network can seriously disrupt the packet forwarding services for the neighboring ASes and the remote DDoS victim networks. In both attacks, we use the same example as in Section I, i.e., the filtering network manipulates the DDoS-victim-submitted filter rule $R=$[Drop 50% of HTTP flows destined to victim network].

[Goal 1] Discriminating neighboring ASes.

ASes expect their traffic to be reliably forwarded by the transit networks. Yet, when a transit network offers the filtering services for the sake of DDoS defense, the packet forwarding service can be silently degraded differently for different neighboring upstream ASes. In particular, the filtering network can apply the modified filter rules for each neighboring AS based on its own business preference. Instead of applying the same original rule $R$, the filtering network may apply $R^{\prime}_{A}=$[Drop 20% of HTTP flows destined to victim network] for traffic flows delivered by AS $A$ and $R^{\prime}_{B}=$[Drop 80% of HTTP flows destined to victim network] for traffic flows delivered by AS $B$. Such discriminatory filtering is hard to detect because individual neighbor AS $A$ and $B$ do not know the unmodified rule $R$ and, even if they know $R$, they cannot determine if the traffic filter applied to their packets is $R$. Each neighbor AS may try to infer the packet drops of the end-to-end path indirectly (e.g., via monitoring TCP sessions); yet, it is insufficient because pinpointing exact location of packet losses (a.k.a. fault localization) is known to be hard without large-scale network collaboration [8, 12].

[Goal 2] Reducing operational cost with inaccurate filtering.

To reduce the operational cost, the malicious filtering network may violate the filter rules submitted by the DDoS victims. For instance, consider that the malicious filtering network wants to use only 10 Gb/s of its filtering capacity for the rule $R$ while total incoming traffic of 50 Gb/s should be sent to the filters. To achieve this, the filtering network can send only 10 Gb/s traffic to its filters and execute the unmodified rule $R$. For the rest of the 40 Gb/s traffic, however, the filtering network can simply allow or drop all without using the filtering capacity. Since the victim network has no information about the incoming traffic arrived at the filtering network, it cannot directly detect this attack.

II-B Desired Properties: Filtering Verifiability and Scalability

As we introduced in the Introduction, we have the two desired properties of the VIF design. First, to remove the attack capabilities required for the filter-rule violation attacks, VIF desires to have the filtering verifiability. A filter rule $R$ is said to be verifiable if any modification of the rule $R$ and its execution by the filtering network can be detected by the victim networks or the direct neighboring ASes. Second, our VIF system must be easily scaled up because DDoS attacks are increasingly scalable in both volume and complexity. We have observed an escalation in the volume of DDoS attack traffic [68, 73] and attacks are getting more sophisticated; e.g., multi-million bots are becoming more common [5].

II-C Trusted Execution Environment with Intel SGX

VIF uses TEEs, particularly Intel SGX in this work, as the feasible hardware-based root of trust. Intel Software Guard Extensions (SGX) is a recent architectural feature that allows secure execution of a program on a computing infrastructure in control of an adversarial operator [20, 48]. SGX also supports secure execution of a user-level program with no modification of underlying commodity software stacks [65, 13, 11]. In particular, it offers the isolated execution of the application logic in a protected memory region, called an enclave, which prevents the operator from tampering it. Moreover, it supports remote attestation that allows a third party to audit that the correct application and data has been loaded in an enclave. The attestation process starts when a verifier issues an attestation challenge to the enclaved application. The enclave then provides a report, which is cryptographically signed with the attestation key of the SGX hardware. Next, the attestation report is verified by the Intel Attestation Service (IAS), which is distributed globally [38]. Recently, Intel also allows anyone, who gets a certificate from Intel, to run their own remote attestation services and verify the attestation report [61].

II-D Assumptions

We assume that ISPs (e.g., victim networks) trust the remote attestation process for the integrity guarantees of the VIF enclave. We also assume an idealized implementation of VIF that has no backdoor. The full implementation of VIF is open-sourced for public scrutiny222VIF is open sourced at https://github.com/InNetworkFiltering/SGX-DPDK. We leave a formal verification of VIF implementation as future work. We consider all hardware and side-channel attacks (e.g.,[31, 44, 36]) as out of scope of this paper since countermeasures to these (e.g., [17, 32, 63, 64, 21, 67]) are orthogonal to the design of VIF.

III Auditable Filter Design

The VIF filtering operation is enclosed by an SGX enclave where the integrity of its execution is guaranteed, i.e., a malicious filtering network cannot tamper it. Furthermore, the filtering internal logic and states are also securely verified via the remote attestation process [38]. The isolated execution and remote attestation are useful in realizing the auditable filter; yet, they are insufficient because (1) the filtering decisions can be influenced by the external inputs to the filter such as packet order and time clock feeds, which are controlled by the filtering network; and (2) the malicious filtering network may redirect the traffic within its network to bypass the filtering operations.

To address these two challenges, we propose the filtering operations to be stateless and hence be independent from the external inputs (§III-A) and implement the enclaved packet logs to detect bypassing attempts (§III-B).

III-A Stateless Filter Design

To analyze the dependencies of the enclaved filters, let us first describe an abstract model for our enclaved filter $f$:

[TABLE]

where $\langle{p_{i},a_{i}}\rangle$ denotes that packet $p_{i}$ arrives at the enclaved filter at time $a_{i}$ (measured by the enclave’s internal clock), $\langle{p,a}\rangle$ represents the packet $p$ that is being evaluated and its arrival time $a$, and the following time relationship holds $a>a_{1}\geq a_{2}\geq\cdots$.

Notice that in this abstract model, the filtering operation of a packet $p$ depends on the packet arrival time and the order of the packets, which can be exploited by the filtering network (see next). Here, we summarize the two properties that are needed to make VIF filter auditable:

• $\bullet$

Arrival-time independence. The filtering decision should be independent of packet-arrival time because it can be easily manipulated by a malicious filtering network (e.g., delaying individual data packets). Moreover, a malicious filtering network can delay the time query/response messages to/from the trusted clock source for the enclave [4], slowing down the enclave’s internal time clock.

• $\bullet$

Packet-injection independence. The filtering decision should not depend on the previous packets since a malicious filtering network can also inject any arbitrary packets into the traffic flow and influence the filtering decision.

Thus, to ensure that the filtering operations are auditable, the filtering function $f$ can be independent of all the previous packets and their arrival times; that is,

[TABLE]

which simplifies the filter design to $n$-tuple (e.g., srcIP, dstIP, srcPort, dstPort, protocol) per-packet filters. In other words, the filtering decision of packet $p$ solely relies on $p$, such as five-tuple bits. While we acknowledge the limitations of such a simple $n$-tuple filter design (e.g., incapable of handling complicated application-level DoS attacks), we note that it is sufficient for handling large volumetric attacks that are in imminent need of in-network filtering.

Particularly, for handling volumetric attacks, we allow victim networks to express filter rules for exact-match five-tuple flows (e.g., a specific TCP flow between two hosts) or coarse-grained flow specifications (e.g., HTTP connections from hosts in a /24 prefix). Details for several practical design points for our enclaved filter are found in Appendix A.

III-B Filter Bypass Detection

The auditability of the VIF filter guarantees that the filter operates correctly for the given packets from a filtering network to a victim network. However, packets may not be properly filtered when a malicious filtering network configures the traffic to bypass the VIF filter, hence violating the filter rules. Particularly, the manipulation of the traffic flows happen outside of the protected enclave and thus cannot be detected by the auditable VIF filter itself. We categorize the filter bypass attacks as follows:

• $\bullet$

Injection after filtering: The VIF filter drops a packet $p$ but the adversary injects a copy of $p$ into the packet stream that is forwarded to the victim;

• $\bullet$

Drop after filtering: A packet $p$ is allowed by the filter but the adversary drops $p$ before forwarding it to the victim; and

• $\bullet$

Drop before filtering: The filtering network drops a packet $p$ even before it is processed by the filter.

Note that, we do not consider injection before filtering operations by a filtering network as an attack because it does not affect the filtering decision due to the packet-injection independence property of the filters (see Section III-A).333Moreover, the detection of packet injections before the enclave operations is hard without explicit coordination with traffic sources.

Bypass Detection.

We allow the victim network and the neighboring ASes of the filtering network to detect such bypass attempts by implementing the accountable packet logs inside the enclave for incoming and outgoing packet streams, see Figure 2. For each packet log, we utilize a sketch, particularly a count-min sketch, a memory-efficient data structure that stores summaries of streaming data [19]. With the sketch-based packet logging, the VIF filter keeps only the measurement summary inside an enclave and significantly minimize the memory footprint; e.g., less than 1 MB per each sketch. With some additional data-plane optimizations (see Section V-A), the computational overhead of computing two sketches per packet is negligible (see Section V-B).

To detect bypass attempts by the filtering network, the victim network queries the authenticated outgoing packet logs from the VIF enclave and compares it with its own local sketch. Any discrepancy between the two sketches implies injection after filter and/or drop after filter attacks by the filtering network. The computation and bandwidth overhead for the logs queries is negligible; i.e., sketching is highly efficient and requires sending only a few MB of sketch memory via the already established channel with the victim network. 444The computational overhead of the victim network should also be low since it only requires an efficient sketch on a commodity server without SGX overhead. When the victim network detects any bypass attempt, it can decide to abort the ongoing filtering request with the filtering network. In practice, the VIF filtering network should allow a short (e.g., a few minutes) time duration for each filtering round so that victim networks can abort any further request quickly when it detects any bypass attempts.

Similarly, individual neighbor ASes of the filtering network can detect the drop before filtering attacks by comparing their own local packet logs with the authenticated incoming packet logs of the VIF filter. As a result, the neighboring ASes can choose another downstream network when they obtain the evidence that their downstream network offers filtering services but intentionally drops their packets before they reach the VIF filters.

Handling malicious intermediate ASes.

The bypass detection mechanism may cause false positives when some packets are dropped after leaving the VIF filter but before reaching the victim network. When this happens, the victim network cannot accurately pinpoint where the packet drop has happened. The packet could have been dropped by one of the intermediate transit networks between the VIF filtering network and the victim network, or by the VIF filtering network itself. This, often-called, fault localization is known to be difficult unless all the networks (including the VIF filtering network, the victim network, and intermediate ASes) collaborate [8, 12]. However, such large-scale collaboration is unlikely in today’s Internet.

Therefore, instead of locating such packet drops, VIF allows the victim network to dynamically test all the intermediate ASes by rerouting its inbound traffic to avoid each of ASes being tested in a short time using the well-known BGP poisoning-based techniques (e.g., [39, 66]). Due to the space constraint, we describe the detailed test steps in Appendix B.

IV Scalable Filter Design

In this section, we first analyze the performance (e.g., throughput, network I/O) bottlenecks of a single auditable filter (§IV-A) and then describe a scalable filtering design with multiple enclaved filters running in parallel and an untrusted load balancer (§IV-B).

IV-A Bottlenecks: Maximum Bandwidth and Number of Rules per SGX Filter

Recent works such as mbTLS have demonstrated the 10Gb/s performance per enclave can be reached with a four SGX cores machine [52]. Although the processors with six or more cores available on the market 555List of SGX-enabled processors is available at: ark.intel.com. may support larger bandwidth, we consider the maximum network I/O performance of each SGX enclave is 10 Gb/s in the rest of the paper.

Since the SGX-based filter must match the installed rules with incoming flows to perform filtering, the number of filter rules naturally becomes the bottleneck of the filter’s performance. Indeed, we measure the throughput of traffic processed by a single enclaved filter with different numbers of filter rules and show the results in Figure 3(a). We can see from Figure 3(a) that when the number of filter rules exceeds approximately 3,000, the VIF filter’s throughput performance rapidly degrades.

One of the explanation is that, when the number of filter rules increases, the lookup table for the packet processing inside an SGX enclave also grows accordingly. Even when we use the state-of-the-art multi-bit tries data structure for looking up the filter rules (see Section V for details), the memory size of the lookup table still grows linearly with the number of filter rules, as shown in Figure 3(b). This result also confirms the Enclave Page Cache (EPC) limit is around 92 MB, as seen in many other works (e.g., [42]).

IV-B Scalable Filtering with Multiple SGX Filters

Given the architectural limitation of secure computing resources in currently available SGX architecture, the single-enclave filtering deployment may not be able to deal with the increasing attack volume and complexity. Hence, we propose a generic VIF architecture that can easily scale up as the number of filters grows, as shown in Figure 4. The scalable VIF design includes multiple enclaved filters running in parallel and some untrusted facilitating components such as the high-bandwidth switching fabric and the controller.

In the followings, we describe how filter rules are distributed and dynamically adjusted among multiple enclaves and explain why a malicious filtering network cannot exploit the untrusted components to manipulate the filtering operations.

Filter rules distribution protocol.

Since the traffic flows being filtered are frequently changed, the filter rules also need to be updated and redistributed among filters accordingly. We consider the filter rules distribution happens in rounds, i.e., the entire filter rule set is given and does not change until the next rule reconfiguration is executed. In each round, the filter rules are calculated and redistributed via a simple master-slave topology among multiple enclaved filters. We illustrate the protocol in Figure 5, where we have filter $E_{0}$ as the master node and $E_{1}$, $E_{2}$ as the slave nodes. In particular, when reconfiguration of filter rules is desired (e.g., traffic volume or the number of filter rules handled by a certain filter exceeds a threshold), any enclaved filter may initiate a rule redistribution round and become the master node. Then, all the slave nodes upload their filter rule sets ($R_{i}$ for $E_{i}$) and the array of the average received flow rates of each rule set $R_{i}$ ($B_{i}$ for $E_{i}$) to the master node. The master node calculates re-configured filter rules, which then be redistributed to all the slave nodes and the load balancer. If the calculation requires the changes to the number of enclaves, necessary additional steps (e.g., creating and attesting more enclaved filters) may be required before the rule redistribution.

Filter rules calculation optimization problem.

In each filter rules redistribution, the master node has to allocate the bandwidth and rules to all enclaved filters. We consider the calculation of the optimal rule sets for each filter enclave as solving a mixed integer linear programming (ILP) optimization problem. We assume $k$ filter rules as $r_{i}$ ($1\leq i\leq k$) and the corresponding incoming bandwidth as $b_{i}$ ($1\leq i\leq k$).666We denote $b_{i}$ as the incoming bandwidth measured for a filter rule for easier understanding. In practice, each enclave would produce byte counts without timestamping them because their individual clock sources are untrusted (see Section III-A). The byte counts are then collected in a timely manner and used for the optimization problem. A single enclave has the memory limit $M$ (e.g., 92 MB) and the bandwidth capacity $G$ (e.g., 10 Gb/s) as we have discussed in Section IV-A. Then, we can decide the minimum number of enclaves as needed as $n_{min}=\lceil\max\left({\frac{1}{G}\sum_{i=1}^{k}b_{i},\frac{ku}{M-v}}\right)\rceil$. To allow some space for optimization, the number of enclaves is taken as $n=\lceil\max\left({\frac{1}{G}\sum_{i=1}^{k}b_{i},\frac{ku}{M-v}}\right)\times(1+\lambda)\rceil$ where $\lambda\geq 0$ is an adjustable parameter for additional enclaves. We define real-valued variables $x_{i,j~{}(1\leq i\leq k,1\leq j\leq n)}$ denoting the portion of bandwidth $b_{i}$ allocated to the $j$-th enclave, and binary variables $y_{i,j~{}(1\leq i\leq k,1\leq j\leq n)}$ representing if rule $r_{i}$ is installed on the $j$-th enclave (i.e., $y_{i,j}=1$). Based on the allocation plan indicated by $x_{i,j}$ and $y_{i,j}$, we consider $C_{j}=u\times\sum\limits_{i}y_{i,j}+v$ as the memory cost function, which is a linear function of the number of rules installed (where $u$, $v$ are constants). Also, $I_{j}=\sum\limits_{i}x_{i,j}y_{i,j}$ is considered as the allocated bandwidth. We present the detailed ILP formulation in Appendix C.

Greedy algorithm to calculate filter rules

. Solving the above-mentioned optimization problem is inherently costly when $k\times n$ is large (e.g. $>10K$). Thus, we propose a greedy algorithm (see Appendix D) that finds a sub-optional solution within a reasonably short time period. The high-level intuition of the greedy algorithm is to pre-compute the two parameters—(1) the number of rules per enclave $h$ and (2) the bandwidth quota per enclave $g$—and arrange the rules and bandwidths for the obtained two parameters heuristically.

Detecting misbehavior of untrusted components.

The load balancer and other components outside of the enclave are untrusted and may misbehave. For example, the load balancer may redirect to a filter the traffic flows that do not match with the filter rules assigned for that filter. However, these misbehaviors can be easily detected by each filter by checking if it receives any packets that do not match the rules it receives from the master node. Each enclave detects such load-balancing misbehavior, and reports it to the DDoS victim. Note that if the load balancer drops the traffic flows that are supposed to be redirected to an enclave, it can be detected by the bypass detection of the auditable filters (see Section III).

V Implementation and Evaluation

We implement a proof of concept of the VIF filter using SGX and various optimizations (§V-A) and then evaluate its data-plane (§V-B) and scalability performance (§V-C) for large attack volume and complexity.

V-A Implementation

Overview

. We build the VIF filter as a Linux userspace application with Intel SGX SDK 2.1 and DPDK 17.05.2 for high-speed packet processing. Figure 6 shows the main components of the VIF filter and the minimal trusted computing base (TCB) of code and data inside the enclave, which includes the entire control-plane and the key parts of the data-plane logic (e.g., packet logging and filtering). The control plane performs remote attestation and manages the keys for communication with a DDoS victim. The design of the data plane follows DPDK pipeline model, where three threads (i.e., RX thread, Filter thread, and TX thread) run on individual CPU cores and packets are passed between cores via DPDK lockless rings (i.e., RX ring, DROP ring, and TX ring). Every thread runs a small loop polling the hardware or software buffers in the previous stage, processes a batch of the packets, and passes it to the next stage in the pipeline.

Optimization: Near zero-copy design.

For every incoming data-plane packet, a VIF filter logs the packet, filters it based on the given filter rule set, and logs it again if it is allowed by the filter, as shown in Figure 7. Figure 7(a) shows a naive approach, where a VIF filter makes the entire copy of incoming packets into the enclave and operate these functions over the packet copies inside the enclave. This full-packet copy approach can be considered as the baseline packet processing mechanism of other existing SGX-based middlebox applications (e.g., Tor nodes [40, 65], TLS middleboxes [52, 37, 30], inter-domain routing [18, 41], and IDSs [70, 62, 30]), where secure operations over the full packet bytes are required (e.g., full packet read or encryption). However, this approach may incur too much overhead when performing line-rate processing due to the remaining EPC memory for a VIF filter is only about 92 MB.

We thus minimize the dynamic memory usage and avoid the paging by copying only certain header fields into the enclave, which we call near zero-copy optimization. This allows more memory space for filter rules and the lookup table. In particular, only a fraction of each packet’s header fields (i.e., the five tuple fields, $5T$, and the packet size, $s$) are copied into the enclave memory along with the memory reference ($\ast$) of the packet, as shown in Figure 7(b). 777Such reduction of byte copies is allowed for our auditable filter applications but this does not necessarily apply to any other SGX-based middlebox applications. The copied data $\langle 5T,s\rangle$ represents the packet and is used for the logging functions and the filter operation. The memory reference $\ast$ is used to perform the corresponding operation (e.g., allow or drop) for the packet in the untrusted memory pool.

With the copied five tuple and the size, we first log each packet using a count-min sketch [19] (with $2$ independent linear hash functions, $64K$ sketch bins, and $64$ bit counters) for memory efficient (e.g., 1 MB) per-source-IP counters. The per-source-IP sketch for the incoming packets enables each neighboring ISPs of the filtering network to detect the ‘drop before filtering’ bypass attack discussed in Section III-B. For forwarded packets, we also record another count-min sketch based on the full 5-tuple bits so that the victim network can detect bypass attempts. The latency increased by the two sketch operations is negligible because only 4 linear hash function operations are conducted in the data-plane path. Each counter has 64 bits and takes only around 1 MB EPC memory per instance of the count-min sketch.

Optimization: Reducing the number of context switches

. Another major overhead stems from the context switches when user application calls the enclave functions (ECall) or the enclaved function calls the outside functions (OCall). We address this performance degradation by reducing both types of calls in the filter thread: (1) VIF only needs one ECall to launch the filter thread and initiates its polling; and (2) the filter thread makes no OCalls as the communication with other threads relies only on the software rings.

Trusted computing base (TCB).

Beyond the DPDK library containing about 64K source lines of code (SLoC), our VIF filter contributes to the TCB only 1,206 SLoC which includes the modification of DPDK ip_pipeline (1044 SLoC) and the packet logging and near zero-copy functions (162 SLoC).

V-B Line-rate Data-plane Performance

Testbed Setup

. We test our implementation with two machines: one is a packet generator and one deploys VIF filter. The packet generator has an Intel E5-2630 v3 CPU (2.40 GHz, 8 cores) and 32 GB memory. The filtering machine has an Intel i7-6700 CPU (3.40 GHz, 4 cores) and 8 GB memory. Both have 10 GbE Intel X540-AT2 network cards and run Ubuntu 16.04.3 LTS with Linux kernel 4.10. On the packet generator machine, we use pktgen-dpdk 3.4.2 to generate the traffic saturating the 10 Gb/s link between the two machines.

Throughput Performance.

We benchmark the maximum throughput performance of the filter with the packet size of 64, 128, 256, 512, 1024, and 1500 bytes for three different versions of the VIF filter implementations: (1) native filter without SGX, (2) SGX-based filter with full packet copy, and (3) SGX-based filter with near zero-copy.

Figure 8 shows the throughput performance for varying packet sizes for the three implementations. For the packet sizes of 256 Byte or larger, all the three implementations achieve the full line-rate of 10 Gb/s. With small packet sizes, however, we observe some degradation due to the use of SGX. Particularly, when we make full packet copies for each incoming packet, the filter experiences significant throughput degradation. The near zero-copy implementation demonstrates 8 Gb/s throughput performance even with 64 Byte packets and 3,000 filter rules. Additionally, we present the experiment results of VIF evaluation in packet per second metric in Appendix E.

Latency Performance.

We also measured the latency for the near zero-copy version with various packet size starting from 128 bytes. The results are 34 $\mu$s (128 bytes), 38 $\mu$s (256 bytes), 52 $\mu$s (512 bytes), 80 $\mu$s (1024 bytes), 107 $\mu$s (1500 bytes). All the measurements are average latency over 10-second run with 8 Gb/s constant traffic load, which are reported by pktgen’s latency measurement function.

Connection-Preserving Filtering Performance.

We evaluate the detailed performance of connection-preserving filtering. We present the result in Appendix F.

Remote attestation performance.

Our detailed remote attestation performance can be found in Appendix G.

V-C Scalable Filter Rule Distribution

We evaluate the solving performance of the mixed ILP optimization problem described in Section IV-B with the CPLEX solver in a server-grade machine with 20 cores. We use 3,000 or more filter rules that would cause the throughput degradation of each VIF filter. In this evaluation, we consider that the total traffic rate going through the entire VIF filter is 100 Gb/s. The incoming traffic distribution across the filter rules follows a lognormal distribution.

With the number of rules more than 3,000 and the number of enclaves more than 10, we find that the CPLEX’s mixed ILP solver cannot return the optimal solutions within any reasonable time period. To evaluate the effectiveness of our greedy algorithm in Section IV-B, that is, how close it is to the optimal solutions, we use a small number of filter rules ($10\leq k\leq 15$) and confirm that the difference between the optimal cost function calculated by the CPLEX’s mixed ILP solver and the results from our greedy algorithm is only 5.2%.

We now compare the execution time of the CPLEX’s mixed ILP solver and our greedy algorithm when the number of rules is between 5,000 and 15,000 and show the results in Table I. To measure the execution time of the CPLEX’s mixed ILP solver for our optimization problem, we configure the solver to stop earlier when it finds a first, sub-optimal solution. As shown in Table I, the CPLEX solver even requires about 200 – 1,600 seconds to find the sub-optimal solutions, which are unacceptably slow for the dynamic filter rules redistribution operations. On the other hand, our greedy algorithm runs three orders of magnitude faster than the CPLEX solver with the same number of filter rules.

Figure 9 shows the extended experiments on the execution times of the greedy algorithm for varying number of rules at a much larger range. We also run this experiment with the total traffic bandwidth of 500 Gb/s, which follows the log-normal distribution. In all the range we test (10K–150K filter rules), the greedy algorithm requires no more than 40 seconds. This enables a near real-time dynamic filter rule re-distribution for large numbers of VIF filters.

VI Practical Deployment at IXP

VIF has a generic architecture designed for any transit networks (e.g., Tier-1 or large Tier-2 ISPs); yet, as the first deployment model, we suggest to deploy it in major Internet exchange points (IXPs). In this section, we present why IXPs, among other transit infrastructure, are the ideal places to introduce our verifiable in-network filtering (§VI-A). We also provide a deployment example of VIF at an IXP (§VI-B) and then evaluate the effectiveness of VIF at IXPs against DDoS attacks with two real attack source data (§VI-C). Finally, we provide a simple cost analysis for deploying VIF service at an IXP for filtering up to 500 Gb/s traffic (§VI-D).

VI-A Internet Exchange Points (IXPs)

IXPs are physical locations where multiple autonomous systems (ASes) peer and exchange traffic and BGP routes. Essentially, an IXP is a large layer-2 fabric and connects ASes (e.g., ISPs, content providers, cloud providers) in close proximity. IXPs provide great convenience to ASes in making peering relationship with many (e.g., hundreds or thousands) other ISPs without the hassle of individual bilateral agreements. The Internet currently has more than 600 globally distributed IXPs [3] and some large IXPs serve multi-Tera b/s traffic volume, which is comparable to large Tier-1 ISPs [7, 15].

Recently, as more video content providers and large cloud providers rely on IXPs for lower cost but faster transit of their traffic, emerging value-added services are expected from IXPs; e.g., application-specific peering, inbound traffic engineering, wide-area server load balancing, and redirection through middleboxes [35]. New innovation for these value-added services has been possible because IXPs have a flexible architecture model (especially, compared to traditional transit networks, such as ISPs). An IXP usually consists of a single data center in a single physical facility; thus, software-based architecture available for data centers can be easily adopted; see software-defined IXPs [35, 33, 34].

Due to their topological centrality, however, IXPs unfortunately often suffer from collateral damage when other networks are targeted by DDoS attacks [57]. Worse yet, IXPs are sometimes directly targeted by DDoS attacks; see an attack incident in 2016 against multiple IXPs: AMS, LINX, and DE-CIX [14]. A traffic filtering service could easily be a natural next value-added service for IXPs [23].

VI-B Deployment example at IXP

We consider the VIF IXP has a generic architecture that includes a layer-2 switching fabric, a route server (which is not highlighted in our paper), and a logical central controller for software-defined switches [35, 34]. Figure 10 illustrates a deployment example of VIF at an IXP. The filtering IXP sets up one or more commodity servers with SGX support. When a victim network is under DDoS attack, it contacts the controller of the VIF IXP via an out-of-band channel.888We assume this channel is available even when the victim network is under attacks. ISPs traditionally have maintained out-of-band channels (e.g., external email servers, telephone lines [51]) for inter-ISP communication. As suggested in [58], the victim network can easily authenticate to the IXP via Resource Public Key Infrastructure (RPKI) [43]. The victim network asks the filtering IXP to create one or more SGX filters and audits it by receiving the validation attestation report(s). After being convinced that the filters have been set up properly (i.e., the remote attestation is done), the victim network establishes a secure channel with the enclaves (e.g., TLS channels) and submits the filtering rules. The load balancing algorithm at the IXP controller receives the rules from the filter parallelization (see Section IV-B) and accordingly controls the switches to distribute traffic destined to the victim network to the enclaved filters. The VIF IXP eventually learns and analyzes all the rules in this step. Finally, the enclave filters perform packet filtering based on the submitted rules and forward the allowed traffic to the victim network.

VI-C Effectiveness of VIF at IXPs against DDoS Attacks

We analyze how much DDoS attack traffic can be filtered by VIF at IXPs with two real attack source data: 3 million vulnerable open DNS resolver IP addresses [72] and 250 thousand Mirai bot IP addresses [5].

Simulation setup.

In our inter-domain routing simulation, we use the CAIDA Internet measurement data with the inferred AS business relationship [2] and the peering membership of world-wide IXPs [3]. We randomly choose 1,000 Tier-3 ISPs as the DDoS victims and consider that each victim receives attack traffic from all the attack sources (e.g., open resolvers and bots) in each case. To determine a traffic forwarding path between autonomous systems (ASes), we assume that each of them applies the following widely adopted BGP routing policies in order [29, 27]: (1) the AS prefers customer links over peer links and peer links over provider links; (2) the AS prefers the shortest AS-path length route; and (3) if multiple best paths exist, the AS uses the AS numbers to break the tie.

We assume that the victim network establishes VIF sessions with several largest IXPs (e.g., the biggest IXPs in each of the five regions, as shown in Table III, see Appendix H). We compute the ratio of flows from the attack IP addresses to the victim network that are handled by at least one of the established VIF filters at the selected IXPs. A traffic flow is said to be transited at an IXP if it traverses along an AS-path that include two consecutive ASes that are the members of the IXP.

Results.

Figure 11 shows how many attack flows can be effectively handled by the in-network filters if installed in some large IXPs. The box-and-whisker plots show the distribution of the ratio of handled attack IPs when Top 1–5 biggest IXPs in the five regions (thus, 5–25 IXPs globally in total) perform in-network filtering service for DDoS defense. In each plot, the solid lines represent the first and the fourth quartile of the data set and the ends of the whisker indicate the 5th- and 95th-percentiles. Also, the red band inside the box represents the median.

Even when a single IXP in each region (thus, total five IXPs worldwide) adopts the VIF filters, the majority of both attack sources (e.g., vulnerable resolvers, botnet) are handled by the VIF IXPs. Approximately, 60% of attack mitigation is expected for the median cases, and 70-80% mitigation can be achieved for the top quarter cases. As more IXPs adopt the VIF filters, even more effective mitigation is achieved. Particularly, Top-5 IXPs per these regions appear to be sufficient enough to offer more than 75% attack mitigation for the median cases, and 80-90% of attack mitigation for the upper quarter cases.

VI-D Deployment Cost Analysis

Let us provide a ballpark estimate of the cost of deploying VIF at an IXP to handle 500 Gb/s of traffic. Note that the 500 Gb/s filter capacity at a single IXP appears to be sufficient because the attack volume at each IXP can be much lower than the aggregated volume at the victim network. For instance, it would require only a few VIF IXPs with similar capabilities to mitigate the biggest DDoS attack ever recorded with 1.7 Tb/s attack traffic [71].

Our experiment in Section V-B shows that a near full line-rate performance of 10 Gb/s per server with four SGX cores is easily achieved. Thus, to handle 500 Gb/s attack traffic, an IXP needs to invest in 50 modest SGX-supporting commodity servers, which would require only one or two server racks. With a commodity server cost is approximately US$2,000, the filtering IXP only needs to spend for one-time investment for US$ 100K to offer an extremely large defense capability of 500 Gb/s. The capital expenditure can be borne by the member ASes (hundreds or thousands) and/or can be amortized by the service fees if the filtering service is economically compensated by the payment from the victims [58]. A rigorous economic analysis of VIF operations in IXPs is out of the scope of this paper and is left for future work.

VII Discussion

We discuss several commonly asked questions regarding the designs and usages of the VIF system.

What if victim networks cause denial-of-service by blocking arbitrary packets?

Malicious victim networks cannot exploit VIF and launch new DoS attacks because filter rules are first validated with RPKI.

Can victim networks exploit VIF?

Even when it is not under attack, an alleged DDoS-victim network can make use of VIF to filter some flows and reduce its operating cost. Regardless of the underlying motivation of VIF requests, however, we do not see this as a serious threat to the VIF system model. First, a VIF filtering network may also save its operating cost due to the reduced upstream bandwidth usage. If not, a VIF filtering network can always refuse to serve any VIF request. Second, a victim network would request filtering of flows that otherwise would be filtered by itself anyway. Third, if payment channels exist between the two networks [58], this is not considered as the abuse of the service but a legitimate outsourced filtering.

Incrementally deployable?

Incremental deployability of new services is a perennial problem in a global Internet. IXPs have been recently highlighted as the practical Internet infrastructure where emerging services can be deployed and used incrementally [35, 34]. A VIF instance installed at even single IXP can readily help reduce DDoS traffic by the direct victims with strong verifiability property.

Can VIF handle any DDoS attacks?

We do not claim that in-network filtering can handle all types of DDoS attacks. VIF demonstrates that it can effectively handle volumetric attacks. However, VIF may not be as equally effective against non-volumetric attacks (e.g., application-layer attacks), and thus we suggest that VIF is used as an additional layer of defense mechanism, which is orthogonal to existing cloud-based defense solutions.

What does a DDoS victim do when misbehavior is detected?

In VIF, we suggest two networks create a filtering contract only between them. Therefore, any one of them can abort the temporary contract whenever one party detects misbehavior.

VIII Related Work

Network DDoS attacks and defenses have been extensively studied in the last 2–3 decades [50]. Here, we summarize a few categories of DDoS defenses and related projects.

VIII-A In-network Filtering

The idea in-network filtering has been the core idea of many DDoS mitigation proposals. There are two prominent approaches to implementation: dynamic filtering and capability-based approaches. Dynamic filtering suggests that the destination ISP requests the ISPs on the forwarding paths to install filter rules at the time of attack, for instance as proposed in Pushback [47], D-WARD [49], AITF [9], StopIt [45]. Capability-based approaches embed capabilities in the packet flows, which can be controlled by the destination hosts to authorize flows in upstream, as proposed in SIFF [74], TVA [75], and Portcullis [53].

Closest to our work is the recent SENSS defense architecture by Ramanathan et al. [58]. SENSS proposes to install DDoS-victim submitted filters at a small number of major ISPs. Ramanathan et al. show that in-network filtering at only four major ISPs in the US would have stopped the Dyn attack happened in 2016 [77]. SENSS also suggests an automated payment channel between a DDoS victim and a filtering ISP so that the ISP can get compensation for the extra filtering tasks. Although the idea is solid and evaluation is promising, the SENSS proposal lacks the filtering verifiability and thus allows several undetectable misbehaviors of the filtering ISPs.999Ramanathan et al. [58] sketch a reputation-based mitigation, which can be used together with our VIF proposal.

Unlike previous in-line filtering proposals, our system VIF focuses on the highly desired but yet-unaddressed security property for in-network filtering system, i.e., the verifiable filter, and demonstrates its feasibility and scalability.

VIII-B Network Function Virtualization with Trusted Hardware

We categorize some of them:

• $\bullet$

Middleboxes: Various network middleboxes have been tested with the SGX capability. TLS middleboxes [52, 37] demonstrate that an SGX-protected middlebox can handle thousands of TLS sessions without violating their end-to-end encryption. ShieldBox [70] and Trusted Click [22] demonstrate that SGX can protect the Click modular router to implement various trusted network functions. S-NFV [62] also discusses general policy, data privacy issues of network functions. LightBox [24] demonstrates the line-rate performance for simple secure packet processing functions. Snort-SGX [42] also demonstrates line-rate performance of Snort 3 along with a DPDK network layer. SafeBricks [55] implements a highly modularized and safe network function architecture in the Intel SGX platform.

Our main contribution is not merely an integration of a simple flow filter function and an SGX architecture but more on addressing scalability and filter-rule violations at network layer that are specific to verifiable in-network filtering defense systems.

• $\bullet$

Privacy-preserving systems: Several systems demonstrate that SGX-based network functions can improve the privacy of anonymity systems: SGX-protected Tor nodes [40, 65],

• $\bullet$

Inter-domain routing: Also, several secure inter-domain routing applications have been proposed to exploit the security guarantees of Intel SGX [18, 41].

• $\bullet$

Verifiable accounting: There also have been some prototype systems that enable the outsource network functions to securely measure the amount of resources used for the requested tasks (e.g., [69] in an SGX platform, [16] in a TPM platform).

With VIF we investigate a unique design point of auditable traffic filters. Particularly, our contribution of VIF is in the design of auditable filters that can handle ever-growing large-scale attack volume and complexity with the SGX support.

VIII-C Cloud-based DDoS Mitigations

The predominant DDoS defense in practice today is an overlay-based filtering approach, such as cloud-based scrubbing services, that performs outsourced filtering in a third-party network on behalf of the DDoS victims (e.g., AWS-Shield [1], Radware DefensePro [6]). Overlay-based filtering approaches are popular particularly because they require no changes to the current Internet architecture. Recent works have proposed advances in such overlay filtering using middleboxes [46, 28] and proposals have discussed large-scale filtering locally at ISPs [25]. However, end users are not satisfied with the status quo. Reports suggest that relying on third-party providers centralizes the DDoS marketplace [28]; costs for small and medium-size victims are high and services are left to the discretion of large service providers [73, 28].

Unlike these proposals, VIF does not rely on the cloud or the local victim network’s capability but directly establishes filtering rules at the transit networks.

IX Conclusion

In-network filtering has numerous known advantages over other proposed DDoS defenses, but the deployment of any of its variants has been stifled. Our proof of concept VIF system addresses one of the core, but largely neglected, problems of source filtering—lack of filtering verifiability—and demonstrates that verifiable in-network filtering is indeed possible with a practical hardware root of trust support. We hope that our study renews discussion on the deployment of in-network filtering in the IXPs and encourages more sophisticated yet auditable filter designs, such as stateful firewalls.

Appendix A Flow-aware Filter Design

We consider deterministic and non-deterministic filter rule types:

• $\bullet$

A deterministic filter rule defines a static {ALLOW, DROP} decision for a specified flow; and

• $\bullet$

A non-deterministic filter rule expresses only the static probability distribution ($P_{\tt ALLOW}$, $P_{\tt DROP}$, where $P_{\tt ALLOW}+P_{\tt DROP}=1$) for a specified flow and the final filter decision for each exact flows is made by the VIF filter. We guarantee the connection-preserving property in VIF filters so that all the packets in a TCP/UDP flow are allowed or dropped together.

The execution of non-deterministic rules with the connection-preserving property can be implemented in two different ways:

• $\bullet$

Hash-based filtering. For each incoming packet $p$, we compute the cryptographic hash (e.g., SHA-256) of its five-tuple bits and the enclave’s secrecy to make filtering decision based on the given probability distribution. For example, packet $p$ is allowed if $H(\texttt{five-tuple-bits}||secrecy)<(2^{256}-1)\times p_{\texttt{ALLOW}}$ with $H(\cdot)=$SHA-256; and

• $\bullet$

Exact-match rule filtering. For each TCP/UDP connection, the filter installs an exact-match rule with a filtering decision randomly chosen based on the given probability distribution.

Note that the two design points have different advantages and disadvantages. The hash-based filtering has a smaller memory footprint for lookup table but it incurs per-packet additional latency for cryptographic hash operations. In contrast, the exact-match filter design tends to have shorter per-packet processing time since it executes only one lookup but it requires a larger memory footprint for lookup tables and adds latency for frequent lookup table updates. We propose a hybrid design where hash-based filtering is performed for new flows until these new flows are installed with exact-match rules at every rule update period (e.g., 5-40 seconds).

Appendix B Dynamic Test of Intermediate ASes

A victim network finds and avoids suspicious ASes that drop the VIF-allowed packets. Particularly, we utilize the well-known BGP poisoning-based inbound rerouting techniques (e.g., LIFEGUARD [39] and Nyx [66]) to reroute (or detour) inbound traffic and avoid traversing any intermediate ASes for a short period of time (e.g., a few tens of seconds). These BGP poisoning-based rerouting technologies do not require any inter-AS coordination and thus the victim network can independently test if any intermediate ASes drop packets (even without the VIF IXP’s agreement).

If misbehavior of an intermediate AS is detected by the victim network, then the misbehaving AS can be avoided for the extended period of time (at least during the VIF session) for auditable filtering. Or, if the victim network continuously witnesses that VIF-allowed packets are dropped continuously when dynamically changing the inbound routes, it may conclude that the VIF IXP itself has been misbehaving. The victim network can then discontinue the VIF contract with the VIF IXP at its discretion.

Note that we do not consider extremely adverse network adversaries, such as dropping all the packets between the VIF IXP and the victim network, which cannot be handled properly by any possible defenses in the current Internet architecture.

Appendix C ILP Formulation for Multi-Enclave Optimization

Our goal is to fully utilize the available resource on $n$ enclaves in terms of the bandwidth and memory, without triggering the performance degradation. Also, the load on each enclave should be balanced, in order to reduce the chance of any enclave getting closer to the limit of $G$ or $M$. Hence, the maximum $C_{j}$ and maximum $I_{j}$ should all be as small as possible, as shown in Equation 3. Note that a constant coefficient $\alpha$ is used to balance two maximums in the sum. Because of the capacity limit of a single enclaved filter, any filter should have less than $M$ memory consumption (Equation 4) and less than $G$ bandwidth load (Equation 5). Since $b_{i}$ is distributed to multiple filters, so their allocated bandwidth with respect to rule $i$ should sum up to the value of $b_{i}$ (Equation 6). Two decision variables are not independent since when $y_{i,j}=0$, $x_{i,j}$ should never be a positive value. (Equation 7).

Appendix D Greedy Algorithm for Scalable Filter Design

The pseudocode in Algorithm 1 summarizes the greedy algorithm we use for the filter rule distribution problem for the scalable VIF filter design in Section IV-B.

Appendix E Additional Evaluation of Throughput Performance of VIF

Figure 13 shows the throughput evaluation in packet per second metric. Taking a closer look at the SGX with full packet copy implementation, we notice that the maximum packet processing rate is capped at roughly 6 Mpps, which suggests the inherent capacity limit of the full packet-copy operations. Unlike the full packet copy version, the near zero-copy version shows no such throughput cap in terms of packet per second.

Appendix F Connection-Preserving Filtering Performance of VIF

As we discussed in Appendix A, the two mechanisms for connection preservation (i.e., hash-based filtering, exact-match rule based filtering) have different advantages and disadvantages. Thus, we present a hybrid design for practical operations. For any new flow that does not match any existing exact-match filter rules, the filter allows/drops based on the hash digest of the 5-tuple of the packets and queues this 5-tuple. At every filter rule update (e.g., every 5 seconds)101010The rule update period can be synchronized with that of the rule re-configuration for scalable, multiple enclave operations; see Section IV-B., all the newly received flows since the last update are converted into exact-match rules and inserted to the lookup table. This hybrid design amortizes the cost of lookup table update by batch processing multiple newly observed flows at every update period. Also, it limits the per-packet latency increase due to hash operations since newly observed flows should be the minority in general. Indeed, our experiments on the performance overhead of the use of hash-based filtering with various packet sizes show no performance degradation, except with small packet size.

Figure 14 shows our experiment on the performance overhead of the use of hash-based filtering for varying fraction of incoming packets. Particularly, when the ratio of hashed packets is low (e.g., $<10$ %), we observe no performance degradation in all packet sizes, except the smallest size (i.e., 64 Byte) where up to 25% throughput degradation is measured. We argue that this performance degradation is easily acceptable because in general, the fraction of newly observed flows within a short period (e.g., 5 seconds) would be small. Moreover, the 64-Byte performance degradation in Figure 14 must be the lower-bound result since it assumes that all the packets are 64 Byte short packets.

Table II shows the benchmark on the time taken to insert the batched new exact-match rules to a multi-bit tri-based lookup table. Our test shows that the batch insertion of filter rules is quite efficient and incurs minimal performance overhead even for large batch size; e.g., only 75 milliseconds compared to the 5-second rule update period.

Appendix G Remote Attestation Performance

VIF performs remote attestation for each new enclave that the VIF IXP launches on its infrastructure. Since VIF is expected to operate under DDoS attacks, we want to ensure that the launching of multiple-enclaves on demand does not become the bottleneck for our deployment model. Here, we measure the total amount of time to complete an end-to-end remote attestation process for one enclave. In our micro-benchmark for remote attestation for the conservative performance tests, we set up the filter enclave and the destination on a cloud machine hosted in South Asia, and the IAS service hosted in Ashburn, Virginia, United States. For an enclave binary of size $1$ MB, the platform takes $28.8$ milliseconds and the total end-to-end latency of $3.04$ seconds with a standard deviation of $9.2$ milliseconds.

Appendix H Top Regional IXPs

We use the IXP peering membership from CAIDA [3] to count the number of AS members of each IXP and summarize the top five IXPs in each of five regions (Europe, North America, South America, Asia Pacific and Africa) in Table III.