Increasing broadband reach withHybrid Access Networks

TL;DR

This paper discusses Hybrid Access Networks that combine xDSL and LTE to extend broadband reach cost-effectively, detailing architectures and a practical implementation using Multipath TCP.

Contribution

It introduces the three architectures defined by the Broadband Forum and shares practical insights from implementing a Multipath TCP-based Hybrid Access Network.

Findings

Successful deployment of Multipath TCP in Hybrid Access Networks

Enhanced broadband reach in less densely populated areas

Validation of architectures through real-world implementation

Abstract

End-users and governments force network operators to deploy faster Internet access services everywhere. Access technologies such as FTTx, VDSL2, DOCSIS3.0 can provide such services in cities. However, it is not cost-effective for network operators to deploy them in less densely populated regions. The recently proposed Hybrid Access Networks allow to boost xDSL networks by using the available capacity in existing LTE networks. We first present the three architectures defined by the Broadband Forum for such Hybrid Access Networks. Then we describe our experience with the implementation and the deployment of Multipath TCP-based Hybrid Access Networks.

Full text

I Introduction

Network operators have deployed a variety of fixed access network technologies (xDSL, DOCSIS, FTTx, …) and various wireless technologies (3G, 4G, FWA and soon 5G). Today’s bandwidth hungry applications like video streaming and various cloud services have forced the network operators to increase the capacity of their access networks. During the last decades, various improvements have been brought in \acDSL networks. While the initial deployments offered bandwidths of only a few Mbps, current deployments with VDSL2 and G.Fast have reached hundreds of Mbps or more. Both VDSL2 and G.Fast can reach their peak bandwidth when the households are close to the street cabinet (from hundreds of meters for G.Fast to a kilometre for VDSL2). These technologies are effective in dense areas where fiber has been deployed close to the end-users. However, there are many regions, such as rural areas, where xDSL cannot reach its optimal bandwidth due to signal loss on long copper pairs.

Governments, in Europe, America and Asia have announced ambitious objectives to provide better broadband services to their entire population. In Europe, the objective is to provide 30 Mbps to all Europeans by 2020. In the USA, broadband is defined as 25 Mbps in downstream and 3 Mbps in upstream.

The European Commission defines the \acNGA as fixed-line broadband access technologies capable of achieving download speeds meeting the Digital Agenda objective of at least 30 Mbps coverage [1]. Based on data from the 28 countries of the European Union, DSL is the most widespread non-wireless technology in rural areas with a coverage of 86.3% in 2017, but it is not considered as an \acNGA. VDSL is only available in 32.5% of the rural areas. FTTP, Cable and DOCSIS barely reach 10% of coverage in these areas. On the other hand, \acLTE covers 89.9% of the rural areas across Europe in 2017. The latest measurements reported by the \acFCC in the USA [2] indicate that only 92.3% of the population has access to a fixed terrestrial service at 25 Mbps. At 50 Mbps, this number drops to 90.8% of the population.

To cope with these government objectives, network operators deploy \acNGA technologies in densely populated areas such as cities. However, this deployment takes time and will still require several years to complete. In rural areas, \acNGA technologies are barely economically viable given the population density. This has encouraged network operators to explore alternatives. One of them is to combine a terrestrial broadband access network such as xDSL with a wireless access network such as \acLTE. The combined availability of xDSL and LTE opens new opportunities for the network operators that own these two types of networks. The Hybrid Access Networks discussed in this paper make it possible to combine the existing xDSL network with an existing wireless network such as \acLTE to provide higher bandwidth services. This enables network operators to provide faster Internet access services without the costly fiber roll-outs that are required by solutions such as VDSL2 or G.Fast.

During the last few years, Hybrid Access Networks have moved from lab prototypes to large-scale deployments of standardized solutions. Standards are important for network operators because they allow solutions from different vendors to inter-operate. We first describe in Section II the reference architectures and the key protocols that the \acBBF and the \acIETF have specified to build Hybrid Access Networks. Then, Section III analyses measurements of the key components of such networks and provides feedback on real deployments.

II Hybrid Access Networks standards

Network operators rely on different standardized technologies to offer broadband services to their subscribers. Standards have became a prerequisite for large scale deployments of access network solutions by network operators. The \acBBF initiated architectural work that was finalized in 2016 [3] while the work on the protocols was progressing within the \acIETF. The \acBBF is currently finalizing the Nodal Requirements for Hybrid Access Broadband Networks [4] that leverage recent \acIETF specifications [5, 6, 7].

II-A Hybrid Access Architecture

To understand how the Hybrid Access Networks will be deployed, it is important to first understand their architecture. A classical access network includes subscriber’s clients using Wi-Fi or Ethernet connected to a \acCPE. The latter is then connected to the operator’s network through one of technologies discussed in the previous sections, which provides Internet access. In a typical xDSL deployment, the \acCPE receives an IP adress from the DSL provider. For simplicity, we consider in this section that addresses are allocated to the \acCPE by the network without discussing the differences between IPv4 and IPv6. We use the @ symbol to represent such an address. If the \acCPE is attached to a single xDSL network, then all the Internet packets to/from the client are transmitted over the xDSL link. Some operators have deployed bonded DSL solutions where two DSL lines are terminated on the same \acCPE. This provides higher bandwidth to end-users, but network operators report that it is difficult to deploy such solutions at a large scale because they typically require the two bonded links to be terminated on the same card on the same \acDSLAM. Given these operational problems, many network operators do not consider bonding two DSL lines as a cost-effective solution to provide higher bandwidth services.

Now, let us consider what happens when the \acCPE is attached to both an xDSL and a \acLTE network as in Figure 1. We call such a \acCPE a \acHCPE in this paper. From an addressing viewpoint, there are two possible architectures to connect these \acHCPEs to the network. A first approach is to configure the address allocation function in both the xDSL and the \acLTE network to allocate the same address on both links (i.e. addresses @cpe1 and @cpe2 in Figure 1 are equal). This requires a strong coordination between the address allocation services on both the xDSL and the \acLTE network. In this case, the HCPE is reachable through the same address over the two networks. It can load-balance the packets that it sends over the two access links at any time and the network can also load balance the packets over the two access links. Several load-balancing schemes are possible on the HCPE and in the network. With per-flow load balancing, each UDP flow or TCP connection is associated to one access link and all the packets belonging to this flow are sent over this link. This solution works well, but it implies that a single flow can only use the bandwidth of one access link. If a single flow needs to be able to utilize the bandwidth of the two access links, then the HCPE and the network must use a per-packet load balancing strategy. With such a strategy, different packets belonging to a single TCP connection can be sent over different access networks. However, xDSL and \acLTE networks experience different delays. This implies that the packets sent over the two networks will be reordered before reaching their final destination. TCP reacts to such reordering by retransmitting packets and reducing its congestion window which in the end slows down the data transfer. Thus, although the TCP packets that belong to a given TCP connection can be sent over both the xDSL and the 4G network, in practice doing so results in reduced performance due to TCP collapse.

From an address allocation viewpoint, the simplest solution to connect the HCPE to both the xDSL and the \acLTE network is to allocate a different address on each access link. An important point to note about this address allocation is that network providers usually implement Network Ingress Filtering [8] to prevent spoofing attacks. The consequence of Network Ingress Filtering is that if address @a1 (resp. @a2) has been assigned on the xDSL link (resp. the \acLTE interface), then only the packets whose source address is equal to @a1 (resp. @a2) can be sent over the xDSL link (resp. the \acLTE interface). If the HCPE sends a packet whose source address is @a2 on the xDSL link, it will be discarded by the provider.

Given that two different addresses are assigned to the HCPE, a simple mechanism would be to assign these two addresses to the hosts attached to the HCPE. IPv6 has been designed with this possibility in mind and all IPv6 implementations can associate several IPv6 addresses to a single interface. With IPv4, this is more difficult as DHCP allocates a single address to each host. However, having several addresses per host would not solve our problem. When a host has several addresses, it selects one of them every time it needs to establish a flow. If a host selects the address assigned by the xDSL network for the first packet of a flow, then the entire flow will be transported over the xDSL network. This implies that a given flow will have no way to utilize the two access links simultaneously. Furthermore, if one of the access links fails, then all the flows that were using this access link are broken, even if the other access link is still available. Another issue for network operators is that hosts can autonomously select their access link by selecting their source address. Many network operators view hybrid access networks as a solution to complement the xDSL network with the 4G network and they would like to use the 4G network only when the capacity available on the xDSL network is fully used. If the hosts autonomously decide the access link that they use, it becomes difficult to implement such policies.

Given the importance of hybrid access networks, several solutions have been explored by the \acBBF and the \acIETF to solve the above-mentioned problems. Three of them are defined in the recent BBF WT-378 specification [4]. They focus on three key requirements: $(i)$ supporting per-packet load balancing to be able to efficiently utilize two different access links for a single TCP connection, $(ii)$ assigning a single address to the end-hosts attached to the HCPE and $(iii)$ supporting Network Ingress Filtering. They all rely on the utilization of a special network function, called the \acHAG, that is deployed by the network operator. Figure 1 describes the architecture of such Hybrid Access Networks. We assume a HCPE that is attached to both a DSL and an \acLTE network. It has received one address from each network (@cpe1 was allocated by the DSL network and @cpe2 was allocated by the \acLTE network). A single address is assigned to each connected host. We also assume that the \acHAG is reachable via two addresses (@h1 via the DSL network and @h2 via the \acLTE network).

II-B GRE-based L3 overlay tunneling

The first realization of this architecture relies on \acGRE Tunnels [4, 6]. This solution can be summarized as follows. The HCPE is reachable via two different addresses (@cpe1 and @cpe2 in Figure 1), but those addresses are not directly exposed to the hosts that are attached to it. The HCPE has a third address that it assigns to its attached host. The @cpe1 and @cpe2 addresses assigned to the HCPE are only used by to create tunnels towards the HAG. When a HCPE boots, it creates these two tunnels and advertises its third address to the HAG. When a host initiates a TCP connection, it sends the first packet to the HCPE. The HCPE encapsulates it in one of its \acGRE tunnels and sends it to the HAG. This encapsulation is required to support Network Ingress Filtering. The HAG decapsulates the packet and forwards it to its final destination. The server response reaches the HAG that maps it to its associated HCPE and encapsulates it in one of the tunnels that reaches the HCPE. The HCPE decapsulates the packet and forwards it to the client host. Both the HAG and the HCPE can use per-packet load-balancing to distribute the load over the two access networks. The solution specified in BBF WT-378 [4] includes several extensions to the basic \acGRE encapsulation. First, sequence numbers are added to each encapsulated packet. These sequence numbers are used on the HCPE and the HAG to reorder the packets that are received over the xDSL and the 4G networks since those have different delays. This reordering ensures that all the packets that belong to a single TCP connection are delivered in-sequence, which prevents the TCP collapse that we described earlier. Furthermore, the HCPE and the HAG regularly measure the performance of the two tunnels to dynamically determine the fraction of the packets that must be sent over each tunnel. Additional details are provided in [6]. Somme lessons learned from the deployment of Tunnel-based hybrid access networks have been discussed in \acIETF presentations [9]

II-C L4 Multipath using MPTCP

The two other standardized architectures rely on Multipath TCP [5]. Multipath TCP is a recent TCP extension that enables hosts to use different paths to exchange packets belonging to a single TCP connection. Multipath TCP includes several mechanisms that enable hosts to efficiently utilize different networks [10]. A Multipath TCP connection starts with a three-way handshake like a regular TCP connection. To inform the server of its willingness to use Multipath TCP, the client inserts in the first SYN packet the MP_CAPABLE option. This option contains a 64 bits key that is associated to this specific connection. The server replies with a SYN+ACK that also contains an MP_CAPABLE option. The client replies with an ACK packet that also contains the MP_CAPABLE option [5]. At this point, the Multipath TCP connection is established over the path used by the SYN packets and both the client and the server can send data. Multipath TCP supports an ADD_ADDR option that enables a host to advertise its other addresses to the remote host. To use another path, the client or the server must create a TCP subflow along this path. This is done by sending a SYN packet that contains the MP_JOIN option. Thanks to the contents of this option, the server can associate the subflow establishment attempt to an existing Multipath TCP connection and authenticate it during the three-way handshake that creates the subflow. Once the subflow has been established, data can flow over any of the available paths. Multipath TCP uses coupled congestion schemes that enable it to react efficiently to congestion on the different paths [10]. Furthermore, Multipath TCP uses its own sequence numbers to reorder the packets sent over different paths at the receiver but also to allow packet reinjection from one path to another one, for example upon timeout on the initially attempted path [10].

With Multipath TCP, a endhost could efficiently use the different paths that exist in hybrid networks. Unfortunately, although several implementations of Multipath TCP exist [11], they are not used on popular endhosts nor on servers. To still benefit from the unique capabilities of Multipath TCP, the \acBBF has decided to rely on Multipath TCP proxies. A Multipath TCP proxy is a network function that converts Multipath TCP connections into TCP connections and vice versa. It can be installed on HCPEs and HAGs. In the implict deployment [4], the HAG is located on the forwarding path between the HCPE and the Internet on the xDSL network (Figure 1). Figure 2 provides a sequence diagram of the establishment of a TCP connection using this implicit deployment. When a host attached to a HCPE initiates a connection, it sends the first packet to the HCPE. The Multipath TCP proxy running on the HCPE intercepts the TCP connection establishment attempt (SYN packet) and replaces it with a Multipath TCP connection establishment attempt (SYN+MPC packet). This packet is forwarded along the DSL network and the HAG intercepts it. The HAG performs the reverse operation and forwards a connection establishment attempt towards the final destination. The destination replies and confirms the connection establishment. This confirmation is intercepted by the HAG that in reaction confirms the establishment of the Multipath TCP connection to the HCPE which eventually confirms the establishment of the connection to the host. At this point, the connection between the host and the server is composed of three parts: $(i)$ a (regular) TCP connection between the host and the HCPE, $(ii)$ a (Multipath) TCP connection between the HCPE and the HAG and $(iii)$ a (regular) TCP connection between the HAG and the server. At that time, the HAG can advertise its address on the LTE network (i.e. address @h2 on Figure 1) by using the ADD_ADDR option. With this information, the HCPE can initiate a second subflow over the \acLTE interface (SYN+MP_JOIN packets exchanged with the HAG). It should be noted that as both the HCPE and the HAG transparently intercept the TCP connections, they do not need to implement any Network Address Translation function. This deployment supports per-packet load balancing as Multipath TCP continuously measures the packet losses and delays over the xDSL and the \acLTE networks and dynamically adjusts the load over the two paths thanks to its congestion control scheme.

The third approach also leverages Multipath TCP, but the HAG does not need to transparently intercept the Multipath TCP connections proxied by the HCPE. The packet flow is illustrated in Figure 3. The HCPE still includes a transparent proxy that proxies the TCP connections established by the local hosts. The main difference with the previous deployment is that the HAG is directly reachable through an IP address over the xDSL network (@h1 in Figure 1). Upon reception of a connection establishment packet from a local host towards a remote server, the HCPE sends a Multipath TCP connection establishment packet towards the address of its HAG and places inside the payload of this packet the address of the remote server (i.e. @s). This leverages the 0-RTT Convert protocol whose specification is being finalized within the \acIETF [7]. The HAG terminates the Multipath TCP connection and immediately sends a connection establishment packet towards the remote server. As in the previous solution, the HCPE can also create a subflows over the \acLTE network to utilize the two access networks. The connection between the client and the server is composed of three connections that are glued together by the transparent proxy running on the HCPE and the 0-RTT convert protocol that is running on the HAG. This is illustrated in Figure 3.

III Implementation and deployment experience

As explained in the previous section, the architecture proposed by the \acBBF for Hybrid Access Networks relies on two components: $(i)$ a Hybrid Aggregation Gateway and $(ii)$ a Hybrid-CPE. We describe in this section our experience in implementing and deploying those two components and evaluate their performance.

III-A Hybrid CPEs

A HCPE is defined as a CPE which is attached to both an xDSL and a 4G network. Most of the deployed xDSL and 4G CPEs use Linux. We have extended several of these with the open-source Multipath TCP implementation in the Linux kernel [12]. This stack completely supports the Multipath TCP protocol and is considered to be its reference implementation. We extend it with a transparent proxy that efficiently moves data from one Multipath TCP connection to one TCP connection and vice-versa. These data transfer operations are implemented in the Linux kernel for performance reasons [13].

One HCPE typically serves only a few clients in a home network, connected through Wi-Fi or Ethernet. Performances are still important because the CPU of the CPE has other services to run besides the MPTCP proxy. Some deployments rely on a HCPE that supports both xDSL and \acLTE while others use a regular xDSL CPE that is attached over a Gigabit Ethernet interface to an \acLTE CPE. This simplifies the deployment as existing xDSL CPEs can be reused.

The maximum throughput that a HCPE can sustain mainly depends on the performance of its CPU. Older CPEs can reach a throughput of 100 Mbps while more recent ones such as most 4G gateways can reach much higher throughput. As an example, we evaluated the performance of a simple off-the-shelf CPE: the Linksys WRT1200AC. This CPE is equipped with a Marvell Armada 385 CPU clocked at 1.33Ghz. We attach a client running iPerf to this CPE and emulate the two access networks by using Gigabit Ethernet links, with different delays. A 20ms delay on one link to emulate the DSL network, and between 20ms and 80ms to emulate the LTE network. The difference of delay between the two link will have an impact on the reordering of the packets. We use iperf to generate ten downlink TCP connections that transfer data during 30 seconds. The baseline throughput for the setup on a Gigabit Ethernet link with 20ms delay is 936Mbps, with a total CPU usage of 70%. Figure 4 plots, in plain lines, the iperf throughput measured on the client in function of the bandwidth available on both links, with a symmetric split 50% on each. The values of the maximum bandwidth on the X axis are ranging from 1 Mbyte/sec to 128 Mbytes/sec, converted in Mbps, and have a maximum at 2x512 Mbps, because the client is connected to the CPE with a Gigabit Ethernet link. Measurements are average over 5 iterations, and the standard deviation is indicated in black. Each line corresponds to a different delay on the emulated LTE link, while the delay for the emulated DSL is always 20ms. It shows that the delay difference between the two links has no influence for throughputs below 576Mbps, but the aggregated speed decreases with an increasing difference. It also plots, in dashed line, the average usage of one of the CPU in function of the available bandwidth. We can see that the CPU usage increases with the aggregated throughput, but that a retail CPE can achieve up to 900Mbps by combining two 512Mbps links.

The network operators who have contributed to the standardization of hybrid access networks [4, 3] have insisted on the ability to support specific policies such as specifying the ports or address blocks which can benefit from aggregation, but also ensuring that the \acLTE network is only used when the DSL one is saturated. One key HCPE function is to decide when to create an \acLTE subflow based $(i)$ on operator objective to lower its costs and $(ii)$ user experience optimization, which is best when both networks are used. For subscribers, the experience depends on the available bandwidth that is the combination of bandwidth from the xDSL and \acLTE networks. For the operator, the optimal business objective is to first use bandwidth from the least costly network (xDSL in the case considered here), and then use the minimum necessary bandwidth on the other network. The trade-off between those apparently mutually exclusive objectives is achieved by the overflow concept. The HCPE always starts to use the xDSL network, until the available DSL capacity is fully used, and then overflows on the \acLTE network. The overflow mode leverages two features of Multipath TCP [12, 14]. The HCPE continuously measures the load of the DSL link. If the average load is low, it does not attempt to create subflows over the \acLTE network. If the average load is above a configured threshold (e.g. 80% of the DSL link capacity), then it automatically opens one subflow over the \acLTE network for each proxied TCP connection. We then rely on a specialized packet scheduler to preferentially use the xDSL network. Multipath TCP includes a flexible packet scheduler [14] that selects the subflow that is used to transmit each packet.

This overflow is illustrated on Figure 5 where a hybrid access is simulated with a CPE and a HAG acting as MPTCP proxies (as illustrated on Figure 1), and a client host running iPerf, with the two access networks simulated as Ethernet links between the HCPE and the HAG. We use iperf to generate a single downlink TCP connection that operates at 30 Mbps during 15 seconds. We consider the capacity of the \acLTE link to be unlimited. Figure 5 plots the measured throughput on the DSL link (blue line), \acLTE link (red line) and aggregated (yellow line), averaged on the duration of the iperf test, in function of the theoretical capacity of the DSL link. As expected, the DSL link is at maximum capacity with a measured throughput following the theoretical capacity on the link, and the measured total aggregated throughput is around 30 Mbps. We observe that the measured throughput on the \acLTE link corresponds to the difference between the 30 Mbps of the iperf and what is available on the DSL link. This clearly shows that the Hybrid Access Network utilizes both networks to achieve the required throughput, but utilizes the \acLTE network only when the DSL one is saturated.

III-B Field Deployments

Several network operators have started to deploy Hybrid Access Networks using the Multipath TCP solutions described in this paper. Their main use case is to provide faster Internet access services in rural areas where the length of the twisted pair lines does not allow the xDSL technologies to reach more than a few Mbps. The customer surveys that they have conducted indicate that these Hybrid Access Networks improve the quality of experience perceived by the end-users.

Several deployment models can be used for Hybrid Access Gateways. Ideally, they should be installed at locations where both the fixed and the cellular networks are present. Some operators deploy a small number of high capacity HAGs in their backbone. Others have a more decentralised network where both the fixed and the cellular network coexist at multiple locations. In this case, they deploy HAGs at all these locations. Many operators opt to deploy HAGs running on virtual machines to ease their management and provisioning.

As an illustration, Figure 6 provides some of the statistics collected on one HAG deployed in a European network. This HAG uses the implicit mode and serves about 10k homes. This HAG is running on a server equipped with Intel Xeon-Platinum 8180 (2.5GHz/28-core/205W) and four Mellanox ConnectX-4 10Gbps interfaces. The four interfaces are combined in a 40 Gbps \acLAG and divided in three \acVLANs: one attached to the DSL network, one to the LTE network and one to the Internet through a backbone router.

Figure 6 confirms in a real deployment several of the points that we mentioned earlier. The statistics were collected on a normal week day. The peak hours are during the evening and most of the users download data. The top part of Figure 6 provides the evolution of the CPU load of the server. It increases slowly with the traffic, but reaches less than 25% on this server while serving a bit more than 10 Gbps of Internet traffic. During the peak hours, there were almost 7.5k active users and the HAG maintains on average 180k Multipath TCP connections. During the peak hours, the HCPEs establish up to 6100 MPTCP subflows per second through this HAG. The bottom part of Figure 6 shows the distribution of the traffic over the different interfaces. The HAG receives up to 10.26 Gbps from the Internet and forwards up to 5.79 Gbps over the DSL network and 4.82 Gbps over the LTE network. The distribution between the DSL and the LTE network depends on the profiles of the users and the maximum bandwidth of the served DSL links and the bandwidth cap over the \acLTE connections. We observed in many deployments that the DSL and LTE loads were similar although the LTE connections have a higher bandwidth than the DSL ones.

IV Conclusion

During the last few years, Hybrid Access Networks have matured. The architecture initially proposed by the \acBBF has been realized by leveraging new \acIETF protocols. We have described the key principles behind the tunnel-based and the Multipath TCP-based solutions. The first deployments combine DSL and \acLTE to provide higher bandwidth services in rural areas where the length of the copper pairs does not enable the deployment of faster DSL services such as VDSL2. Our measurements indicate that the Multipath TCP-based solutions, which can be deployed as a software extensions on CPE routers and on commodity x86 servers, enable to use the \acLTE network only once the DSL link becomes saturated while still supporting high bandwidth services.

The ongoing deployments demonstrate that network operators can optimize their assets by combining different heterogeneous access networks to provide faster, cheaper or more resilient services to their customers. Although the first Hybrid Access Networks combined xDSL and LTE, the technology is generic and can be applied to any type of network. Recently, the 3GPP has decided to also reuse the 0-RTT convert protocol to support the Access Traffic Steering, Switch and Splitting (ATSSS) that will enable 5G networks to combine 5G with other technologies such as Wi-Fi or satellite.

Acknowledgments

We would like to thank the Tessares employees who have participated in the development of the hybrid access network solutions described in this paper as well as Bart Peirens, Mohammed Boucadair and Guiu Fabregas for their contributions to the \acIETF and \acBBF specifications.