Scalable Real-time Transport of Baseband Traffic

TL;DR

This paper introduces DISTRO, a scalable real-time baseband transport network for wireless systems like Massive-MIMO, ensuring bounded delays and optimized capacity through a novel Fat-Tree design and analytical methods.

Contribution

It presents a new Fat-Tree-based design called DISTRO that guarantees real-time bounds and capacity optimization for large-scale wireless baseband transport networks.

Findings

DISTRO bounds maximum end-to-end delay for baseband packets

The design achieves scalable real-time performance in Massive-MIMO systems

An efficient algorithm optimizes capacity in the transport network

Abstract

In wireless deployments, such as Massive-MIMO, where radio front-ends and back-end processing are connected through a transport network, meeting the real-time processing requirements is essential to realize the capacity gains from network scaling. While simple forms of baseband transport have been implemented, their real-time analysis at much larger scale is lacking. Towards this, we present the design, delay, and capacity analysis of baseband transport networks, utilizing results from real-time systems in the context of wireless processing. We propose a novel Fat-Tree-based design, called DISTRO, for baseband transport, which is a real-time network that bounds the maximum end-to-end transport delay of each baseband packet. It achieves this by placing design constraints and bounding the queuing delay at each aggregation point in the network. We further characterize the wireless capacity using DISTRO and provide an efficient search algorithm for the design of a capacity achieving baseband transport.

Figures

Tables

Table 1. TABLE I: Simulation parameters

$t_s$	50ns	Edge–src.	10Gbps
$t_p$	10ns	Agg.–Edge	10Gbps
Pkt. size ($B$)	1KB	Agg.–Core	40Gbps
$q$-ary tree	2,3,4	Core–Dest.	200Gbps
height ($h$)	2	Flow rate ($R_i$)	1, 1.5, 2, 2.5 Gbps
Radios/edge	4	Simulation Time	1sec

Equations

$$R_i=2Q_i{f}_i.$$

$$T_i=\frac{B}{R_i}.$$

$$D_i=T_{prot}-T_{proc}.$$

$$C_{j+1}\le C_j/q.$$

$$t_j^q\le (q-1)C_{j+1}.$$

$$T_a$$

$$\le$$

$$=$$

$$\le$$

$$=$$

$$T_a<h(t_s+t_p)+2C_1.$$

$$\mathbf{y}=\mathbf{Hx}+\mathbf{z}+{\mathbf{z}}_Q,$$

$$R(\mathbf{Q})={\mathrm{E}}_{\mathbf{H}}[{\log }_2\text{det}(\mathbf{I}+\rho {\mathbf{\Sigma }}^{-1}\mathbf{H}{\mathbf{H}}^H)]$$

$$\underset{\mathbf{Q}\in {\mathcal{L}}^n}{\max }R(\mathbf{Q})$$

$${\tau }_i(\mathbf{Q})=(T_i,D_i),\forall i,\text{is schedulable}$$

$$\underset{\mathbf{Q}\in {\mathcal{L}}^n}{\max }R(\mathbf{Q})$$

$$(T_i,d_i^{{}^{\prime }}),\forall i\in S_k,\forall k,\text{is schedulable}$$

$$enum([L_{k_1},L_{k_2},\dots ,L_{k_n}])=$$

$$[L_{k_1},L_{k_2-1},\dots ,L_{k_n}],$$

$$\dots ,[L_{k_1},L_{k_2},\dots ,L_{k_n-1}]\}.$$

$$\forall t>0,\frac{C}{t}\left(\sum _{i=1}^n{\lceil (t-d_i)/T_i\rceil }^{+}\right)\le 1,$$

$$\forall t\ge d_{min},\frac{C}{t}\left(1+\sum _{i=1}^n{\lceil (t-d_i)/T_i\rceil }^{+}\right)\le 1$$

$$\underset{1\le m\le n}{\max }\underset{k\le N_m}{\max }{W}_m(k,(k-1)T_m+d_m)\le 1$$

$$W_m(k,x)=\underset{0<t\le x}{\min }\frac{C}{t}\left(k+\sum _{i=1}^{m-1}\lceil t/T_i\rceil \right),$$

$$W_m(k,x)=\underset{0<t\le x}{\min }\frac{C}{t}\left(k+1+\sum _{i=1}^{m-1}\left(1+\lfloor (t-C)/T_i\rfloor \right)\right)$$

Full text

Towards this, we present the design, delay, and capacity analysis of baseband transport networks, utilizing results from real-time systems in the context of wireless processing. We propose a novel Fat-Tree-based design, called DISTRO, for baseband transport, which is a real-time network that bounds the maximum end-to-end transport delay of each baseband packet. It achieves this by placing design constraints and bounding the queuing delay at each aggregation point in the network. We further characterize the wireless capacity using DISTRO and provide an efficient search algorithm for the design of a capacity-achieving baseband transport.

I Introduction

In emerging wireless architectures such as C-RAN [1, 2], and Massive-MIMO [3, 4], a baseband transport network connects radios (front-ends) to a backend processing infrastructure. For instance, in a C-RAN deployment, a fronthaul network carries baseband samples between the remote radios and the baseband processors located in a datacenter. Similarly, in massive MIMO, tens of antennas are connected to a common baseband processor through a transport network. Since the real-time capability of baseband transport network is the key enabler of such architectures, its design determines both the scale and achievable wireless capacity of the deployment.

An instance of baseband transport network was realized in ARGOS [4], where 64 antennas are connected over Ethernet for centralized processing. While ARGOS’ transport design is intuitive, its real-time analysis or applicability in a larger network remains unknown. Clearly, a baseband transport network should support traffic aggregation from the radios, for MIMO processing [3], or for resource pooling in C-RAN [5], where a common compute platform decodes data/signals for multiple base-stations. A tree-based design using aggregation switches fits the criteria, which, coupled with packet scheduling policies, form the basis of real-time transport.

Beyond the primitives of aggregation and real-time delivery, the design of a baseband transport network is ultimately guided by the following key requirements:

Guarantees. The real-time nature of the wireless sample processing imposes stringent constraints on the transport network requiring end-to-end (e2e) guarantees for delay and jitter. For instance, the transport delay bound can be as low as few microseconds for WiFi samples [6], to few hundred microseconds for LTE samples [5]. Moreover, the transport behavior of the radio samples must be predictable, i.e., given a network topology and the traffic sources, one must be able to model the delivery of the baseband traffic. This model is necessary for an e2e schedulability analysis — determining whether the given network can meet the requested delay bounds. However, modeling packet traffic in the network core is known (e.g., from the real-time systems literature) to be intractable due to the non-periodic nature of arrivals [7, 8]. Therefore, in a general baseband transport network consisting of multiple switches, it is not entirely clear which packet scheduling policies (implemented by switches) will achieve e2e schedulability.

Scalablility. Considering the size of future radio deployments, the baseband transport design should be scalable. It must be extensible to any number of radios while preserving its predictable behavior. Also, it must require only few additional resources (e.g., cables, switches) to add a large number of radios to the network. The scalability of baseband transport is desirable, if not necessary, for massive MIMO systems which are equipped with tens or hundreds of antennas/radios. Existing datacenter designs [9, 10] that are optimized for scalability, are likely candidates for baseband transport. However, they are primarily designed to handle bisection traffic between the compute clusters, whereas traffic flows in a baseband network are exclusively in the north–south direction.

Optimality. Whether the baseband traffic is schedulable or not depends on the transport rate, which, in turn, depends on the sample quantization widths used at the radios. As the selected quantization widths affect the wireless capacity through quantization noise [11, 12], there is an indirect dependency between schedulability of the traffic and the wireless capacity of the network. Ideally, the network should operate at a point that ensures schedulability but also maximizes wireless capacity.

To meet the above requirements, we propose DISTRO, a design for real-time baseband transport (such as fronthaul) networks that can potentially scale to a large number of radios. DISTRO utilizes a logical tree structure of radio front-ends and network switches: the radios represent the leaf nodes of the tree, the network switches represent the internal nodes, and the root of the tree is a common aggregation point that connects to a pool of baseband processors. DISTRO’s design supports real-time transport as it allows us to bound the maximum transport delay of each baseband packet. Specifically, using a constrained tree design, the upper bound on the waiting time at any switch can be obtained irrespective of the input arrival sequence, from which one can obtain the maximum total delay of a baseband flow. As a result, the network switches can utilize schedulability results from the real-time systems literature and implement scheduling policies (such as EDF [13] and fixed-priority [14]) to achieve e2e delay guarantees.

Since scheduling policies are subject to the baseband traffic parameters, any addition of radios or changes in them requires policy changes in the entire network, making the design unscalable. Therefore, the tree structure in DISTRO is partitioned into aggregation- and edge-switch networks; the schedulability analysis is done only at the edge-switch network whereas the default packet scheduling (FIFO) is implemented in the aggregation-switch network. This logical division along with the tree-based design enables transport scaling to a large number of radios.

DISTRO selects quantization widths to maximize the wireless capacity while ensuring e2e schedulability. A brute-force search of optimal quantization widths, however, has exponential complexity in the number of radios. By using the monotonic dependence of the wireless capacity and the schedulability on the quantization widths, we propose a greedy-based approach that is optimal but has much lower runtime complexity.

In summary, this paper makes the following contributions:

• •

Design. Proposal of a Fat-Tree architecture for scalable deployment of baseband radios;

• •

Delay. Calculation of the maximum delay bound of a baseband packet in the network and its use to achieve e2e schedulability; and

• •

Capacity. Characterization of the wireless capacity under the schedulability constraint, and development of an efficient search algorithm to maximize the capacity.

The rest of this paper proceeds as follows. Sec. II provides the background on the baseband transport while Sec. III presents our proposed transport design and its evaluation results in a simulated network scenario. In Sec. IV, we obtain the maximum wireless capacity with end-to-end schedulability through our proposed algorithm. Finally, Sec. V presents the related work and Sec. VI concludes the paper.

II Wireless Baseband Transport

In this section we provide the background on baseband transport and processing, and describe how real-time scheduling fits into its design.

II-A Network Primitives

The radio front-ends in a baseband network act as converters between RF samples and complex (I and Q) baseband samples. Fig. 2 shows the components of such a radio. In the receive mode, the RF signal is down-converted, filtered and passed through an analog-to-digital converter (ADC) that gives out a digitized (e.g., 16-bit) stream of baseband samples. This stream is broken into fixed-size blocks, which are then transported as payloads in special-purpose packets generated with appropriate headers and tags.

Suppose there are $n$ radios in the network, where each radio is denoted by index $i\in\{1,\ldots,n\}$. Let $f_{i}$ denote the desired sampling frequency from the ADC (achieved through decimation by digital-down converters), and let $Q_{i}$ be the number of bits used to represent each I (and Q) baseband sample. Then, the transport data rate (in bits/s) of radio $i$ can be expressed as:

[TABLE]

Further, let $B$ denote the fixed payload size (in bits) of a transport packet. The inter-packet arrival time (in seconds) at radio $i$, assuming negligible packet overhead, is given by:

[TABLE]

Since the ADC operates at a fixed frequency, and fixed-size blocks are used, the packet inter-arrival time is a constant at each radio, which we refer to as the period of the arrival process.

Example: USRP [15] is a common software-radio platform that uses the UDP protocol for baseband transport. Fig. 2 shows the timestamps from the transport log of a USRP2 running at 25MHz sampling rate and payload size of 1492 Bytes. Using Eqs. (1) and (2), the packet inter-arrival time with 8-bit quantization is calculated to be $2.98\mu$s. This is indeed close to inter-packet arrival time $\in[3,~{}4]\mu$s observed from the USRP2 logs (Fig. 2). Note that the logged arrival period has a minimum 1$\mu$s resolution.

Wireless protocols have a fixed e2e processing deadline for PHY-layer primitives such as channel sensing and decoding. Assuming a fixed (or worst-case) processing time at the baseband processors, the e2e PHY deadlines impose a maximum transport delay. Therefore, in order to support real-time processing, the generated baseband packets from the radios must be transported to the baseband processors within a fixed amount of time. That is, each radio $i$ has an e2e transport delay bound, $D_{i}$, that the transport network must satisfy.

The traffic specification of radio $i$ is thus given by a 2-tuple $\tau_{i}=(T_{i},D_{i})$, which represents the inter-arrival time, and the e2e delay bound, respectively. The radio traffic is said to be schedulable if for all $1\leq i\leq n$, the maximum delay experienced by packet of radio $i$ is not greater than the requested delay bound $D_{i}$.

The transport network in large deployments of Cloud-RAN runs over a fiber infrastructure such as dark fiber[2]. While in indoor environments, the radios may be deployed using high-capacity Ethernet or Infiniband links [4]. In both scenarios, baseband samples are exchanged through packet transmissions. Thus, the baseband transport network can be modeled as a packet-switch network with one or more network switches. Every packet in the network passes through multiple links, switches, and routers before reaching its destination. While many routes can exist for a packet, we assume fixed routing in the network, which is necessary for e2e delay guarantees [16].

Despite its advantages, packet-switching introduces various delays at each link in a selected route. The e2e packet delay is the summation of delays over links and switches along the selected route, which is composed of:

• •

Propagation delay ($t_{p}$): the time taken for the packet to reach the next switch;

• •

Switching delay ($t_{s}$): the time taken for the packet to move from the ingress to egress port of a switch;

• •

Transmission delay: the time needed to transmit the packet, which is a function of the link capacity; and

• •

Queuing delay($t^{q}$): the waiting time of packet in a switch’s egress queue.

Among the different delays, the queuing delay is the only unknown that can be different for each packet in the network. In general, it is a function of the switch’s scheduling policy and the input arrival sequence. One needs to model these delays at each switching stage, which becomes intractable for large networks as the output sequence from a switch is non-periodic even though packets arrive periodically. For example, consider a simple baseband network (Fig. 3) with periodic baseband traffic from 2 radios having inter-arrival times of $2\mu$s and $3\mu$s, respectively. Assume the processing link capacity is 10Gbps and the packet size is 1000 bytes. That is, the output transmission time of both flows is $1\mu$s. Assume packets from the two radios arrive in the queue at the same time instant in the beginning. Fig. 3 shows the timeline of the queue output. As one can see, the inter-arrival time of the 3.3Gbps flow at the output is non-periodic (inter-arrival times of 2$\mu$s and 4$\mu$s) that is induced from the waiting in the queue. This non-linearity of queuing makes it difficult to model e2e packet arrivals, a fact well-known in the literature [7].

Common packet-switching techniques such as First-In-First-Out (FIFO) and Round-Robin (RR), which are designed for best-effort traffic flows, are not suitable for the real-time traffic that requires e2e delay guarantees. To support real-time baseband traffic, the switches can use various scheduling policies that were developed by real-time systems researchers [8, 7]. Among them, the Earliest-Deadline-First (EDF) scheduling is a natural approach where each arriving packet is assigned a deadline according to the requested delay bound, and the packet with the earliest deadline is transmitted first. This scheduler is optimal [17] in the sense that if packets meet their deadlines using any scheduling policy, so will they using EDF. While the original EDF considered implicit deadlines (deadline is the same as the period) and preemption, one can generalize it further to obtain both necessary and sufficient conditions for schedulability with arbitrary deadlines. Theorem A.1 (Appendix) formally states these conditions for both cases – with and without preemption.

Deadline scheduling is based on dynamic prioritization and thus difficult to realize in practice. A more feasible approach is the fixed-priority scheduling where each traffic flow is assigned a static priority [14] where incoming packets are transmitted in the order of their priority. Theorem A.2 (Appendix) shows that there exists a schedulability test to determine whether the set of traffic flows with given priorities meet their delay bounds. Furthermore, one can do an iterative or offline search and use the schedulability criteria to arrive at the feasible priority-assignment policy if one can be found [18]. The necessary conditions for schedulability, however, are known only under special circumstances (e.g., when the deadline is less than or equal to the period).

II-B Transport Delay Bound

The wireless processing design depends on the wireless protocol and the target architecture. For WiFi signals, where slots are $9\mu$s long, baseband samples are streamed and decoded on the fly [6]. In contrast, LTE has 1$ms$-long subframes that are decoded on an accumulated 1$ms$ buffer of baseband samples [5]. The time required to decode the baseband samples (or frames) depends on the capability and the optimizations of the platform. In this paper, we assume the processing time is fixed, and focus on the transport delays.

The transport delay bound of radio $i$ is computed by subtracting the maximum processing time, $T_{proc}$, from its end-to-end protocol deadline, $T_{prot}$, as:

[TABLE]

In case of WiFi signals, the protocol deadline, $T_{prot}$, can be 4$\mu$ (since CCA assert should occur within $4\mu$s during energy sensing [19]). Assuming $T_{proc}=2\mu$s to perform sample summation, this results in a $2\mu$s delay bound for the transport network. On the other hand, for LTE signals, $T_{prot}=2$ms, as there is no channel sensing, and the protocol deadline is governed by the HARQ process. Consequently, the delay bound is much larger (0.5–0.7 ms) than the WiFi case.

It is worth noting that the transport delay bound, $D_{i}$, is not always fixed but can vary with the number of radios, $n$, since the processing time, $T_{proc}$, typically increases with $n$. For instance, $T_{proc}\propto O(n^{3})$, when decoding $n$ spatially multiplexed signals [20].

III DISTRO DESIGN AND DELAY ANALYSIS

We now describe the construction, requirements, and delay analysis of the proposed real-time transport design for baseband traffic.

III-A Design Philosophy

The design of a baseband network is driven by two key observations. First, the baseband traffic flows exclusively between the radios and the processor pool (cross-traffic between radios is negligible). Second, depending on the application, baseband traffic is aggregated into one or more links for processing. In addition, the baseband transport operates in real time: given the traffic flows and delay bounds, we must be able to give a sufficient, if not necessary, condition to check their e2e schedulability.

We propose DISTRO, a baseband network design that combines the tree structure with real-time scheduling. Fig. 4 illustrates the proposed design in a deployment of heterogeneous radios. The radios are connected to edge switches, and the links from the edge switches are aggregated at multiple levels till the root switch. The destination is assumed to be located at the root switch, which is the common aggregation point for the baseband packets. The traffic destination is assumed to be a physical point that is connected to a pool of baseband processors.

DISTRO’s design accommodates an increasing number of radios without severely affecting their delay performance, and makes the schedulability flexible to the addition and removal of radios. This is achieved by partitioning the baseband network into two components: edge- and aggregation-switch network. The edge switch network contains the edge switches that form the first entry point of a baseband flow. From a deployment standpoint, each edge switch could connect a group of radios that are in physical proximity of each other, for instance, a basestation site in a cellular network. The aggregation network contains all the remaining switches except the edge switches, and its purpose is to aggregate traffic from edge towards the destination.

III-B Design Requirements

The aggregation network in DISTRO is a logical tree of links and switches. To simplify the schedulability analysis, we place the following restrictions on its design.

1) Fat-Tree. A tree is a Fat-Tree if for every switch in the tree, the switch’s uplink capacity is greater than or equal to the sum capacity of the incoming links.

2) Symmetric. A tree is symmetric if for every switch, interchanging the incoming links yields the same tree.

3) Non-preemptive. The network switches always use a non-preemptive scheduling policy, i.e., an ongoing packet transmission is never preempted.

4) Equal packet sizes. The basesband packet sizes in the network are always equal irrespective of their source.

III-C End-to-End Guarantees

As noted earlier, a packet can experience queuing delay at any of the switches in the network. Beyond the edge switch, characterizing the queuing behavior becomes difficult as the packet arrivals are no longer periodic [7]. However, under our symmetric Fat-Tree construction, we can readily bound the maximum queuing time.

Assume $K$ edge switches, and let $S_{i},i=1,2,\ldots K$, denote the set of radios connected to edge switch $k$ where $S_{k}\subseteq\{1,2\ldots,n\}$. For simplicity, assume at time $t=0$ packets from radios of each edge switch are queued at the switch, and the packets arrive periodically at the switch thereafter. Further, assume no transmission or propagation delay from the radio to the edge switch. Let $C_{1}$ denote the transmission time of a packet on the link connecting the edge switch and the next aggregation switch. Since packet size, $B$, is fixed for the network, every baseband packet has the same transmission time going through the edge switch.

Theorem 1**.**

In a $q$-ary symmetric fat-aggregation-tree of height $h$, the radio traffic $\tau_{i}=(T_{i},D_{i}),i=1,2,\ldots,n$, is schedulable, if for every $k,1\leq k\leq K$, the set of traffic flows $(T_{i},d_{i}^{{}^{\prime}}),\forall i\in S_{k}$, with transmission time $C_{1}$ and no preemption, is schedulable at edge switch $k$, where $d_{i}^{\prime}=D_{i}-\frac{1-q^{-h}}{1-q^{-1}}C_{1}-(h+1)(t_{s}+t_{p})$, $\forall i$.

Proof.

We show the proof by construction. Let the transmission time sequence as a packet traverses from the edge to the destination be represented as $C_{1},C_{2},\ldots,C_{h+1}$, where $h$ is the height of the aggregation tree. We denote $h=1$ if the network contains only one aggregation switch.

From the symmetric Fat-Tree definition, it holds that if $C_{j}$ is the transmission time on the incoming link, then the transmission time on the aggregation link, $C_{j+1}$, for all $j,1\leq j\leq h$, satisfies:

[TABLE]

Assume the edge switches use some non-preemptive scheduling policy whereas the aggregation switches use FIFO scheduling. Under non-preemptive scheduling, the inter-arrival time of packets on the outgoing link of the edge switch is equal to or larger than $C_{1}$. Consider the first aggregation switch: the outgoing transmission time is $C_{2}(C_{2}\leq C_{1}/q)$. The maximum queuing time of a packet at the switch is $(q-1)C_{2}$ which occurs when $q$ packets, one packet from each link, arrive at the same time. Note that since the inter-arrival time $C_{1}\geq qC_{2}$, and FIFO scheduling is used, two packets from the same link cannot be transmitted unless queued packets from all the other links are transmitted.

This is illustrated in Fig. 6(a) in the case of two links, where packet $c$ arrives at the same instant as packet $b$, and is blocked by the transmission time of packet $b$.

Continuing along the path in Fig. 5, for every $j$, packets in incoming links with transmission time $C_{j}$ and inter-arrival time greater than $C_{j}$, will have a maximum queuing delay:

[TABLE]

Thus, the maximum total delay of any baseband packet across the aggregation network starting from the edge switch is bounded by:

[TABLE]

where we use the inequality $C_{j+1}\leq C_{j}/q\leq\ldots\leq C_{1}/q^{j}$ using Eq. (4).

Therefore, the aggregation delay bound obtained is independent of the arrivals at in the network. Now, in order for a baseband packet of radio $i$ to meet its e2e delay bound $D_{i}$, the total packet delay at its edge switch must be less than $D_{i}$ less the aggregation delay bound. Adding the switching and propagation delays of the edge switch ($t_{p}+t_{s}$), we arrive at the e2e schedulability of all baseband flows in the network by checking the schedulability of baseband traffic at each edge switch. $\Box$

Theorem 1 provides only a sufficient condition for schedulability, but does not affirmatively tell us if a given set of traffic flows are schedulable. Nevertheless, the result is still useful as it allows us to construct a transport network that guarantees to meet the requested e2e delay bounds. Furthermore, it gives the scheduling policies that the network must implement: non-preemptive scheduling at edge switches, and regular FIFO scheduling at aggregation switches.

The edge switches can implement a non-preemptive scheduler which can either be EDF or fixed-priority. The fixed-priority scheduler is easy to realize with multiple outbound queues at a switch. Each incoming packet is first classified and placed in its corresponding queue, and the queues are dequeued in the order of their priority.

The reason for restricting our construction to a symmetric Fat-Tree and non-preemptive scheduling is simple: the packet transmission (tx) times are equal for the incoming links, and therefore, the maximum queuing delay in the aggregation network is easy to obtain. This is not the case, however, if we assume unequal transmission times, which happens if unequal links are used or when preemption is allowed. For instance, Fig. 6(b) shows the blocking of a smaller packet, $c$, due to the transmission of earlier packets. This example is akin to the head-of-line (HOL) blocking problem that occurs when subsequent transmissions are held up by the first transmission [21]. Then, to arrive at the maximum queuing time in the network, one must examine different arrival sequences and their transmission times at each switch, which can quickly become intractable.

III-D Scalability

The design of DISTRO is scalable on three fronts. First, as the number of radios in the network, $n$, grows, and the number of edge switches, $K$, increases, the maximum total delay of a packet increases only logarithmically. To see this, one can upper bound the total aggregation delay in Eq. (10) as follows:

[TABLE]

The delay bound grows linearly with the height, $h$, which is always less than or equal to $\lceil\log_{2}K\rceil$. For large packets (e.g., Jumbo Ethernet frames), the transmission time is much larger than the switching and propagation time. In this case, the delay scaling factor, $t_{s}+t_{p}$, becomes even less significant.

Second, DISTRO supports heterogeneous baseband radios with differing periods and deadlines. There is no restriction on the type of radio protocol. This aids the deployment of a radio access network over multiple wireless standards, such as LTE and WiFi, at the same physical location.

Finally, each edge switch, $k$, implements a schedulability test locally that is dependent only on the traffic generated from the set of radios, $S_{k}$, connected to it. Any addition or removal of radios requires one to check only local schedulability, without disturbing the performance of other flows. This feature enables incremental deployment of the baseband network.

III-E Run-time Scheduling

The aggregation network in DISTRO implements the default FIFO scheduling. On the other hand, each edge switch implements a non-preemptive scheduler which can either be EDF or fixed-priority. The fixed-priority scheduler is easy to realize with multiple outbound queues at a switch. Each incoming packet is first classified and placed in its corresponding queue, and the queues are then dequeued in the order of their priority.

Background Traffic. Background traffic such as control messages can disrupt the real-time performance of the network. To ensure minimum disruption, we segregate the baseband traffic from the rest of the traffic through flow prioritization. Background and rest of the traffic are placed in a separate queue, which has lower priority than baseband traffic. Since background traffic adds to the non-preemptive delay at each switch, we update the delay bound in Eq. (10) accordingly.

III-F Evaluation

We implement and evaluate DISTRO’s Fat-Tree architecture using NS-3[22]. NS–3 is a discrete-event network simulator that accurately simulates network traffic in large deployments. Table I shows the simulation parameters used in our setup. Each edge switch connects 4 heterogeneous radios that have different flow rates (1, 1.5, 2 and 2.5G) but fixed packet size of 1000 bytes. We simulate a fat-aggregation-tree with three levels: 1) a core switch that is connected to the destination through 200Gbps link; 2) $q$ aggregation switches connected to core switch with 40Gbps links; 3) $q$ edge switches connected to each aggregation switch with 10Gbps links. Thus, the total number of radios in our setup is $4q^{2}$.

We use the NS–3 packet tagging mechanism to tag each baseband packet with a priority level. The priority levels are chosen to achieve e2e schedulability (Sec. III-C). We then implement a packet-classifier at the edge switch that classifies the incoming packet and places it in one of the 4 egress queues. For dequeuing, the queues are searched in decreasing order of their priority. For simplicity, we assume the transport delay-bound is same as the inter-arrival period of each flow. Under this assumption, the priorities are determined by the inverse of the period (equivalent to a rate-monotonic scheduler).

Fig. 7 shows the maximum e2e transport delays and jitter observed in a 3-$ary$ tree with 36 radios. Here, we compare DISTRO’s scheduling with the basic FIFO packet scheduling. As seen from Fig. 7(a), DISTRO meets the e2e guarantees (delay $<$ inter-arrival time) of each flow while the FIFO scheduling misses the delay bound of the 2.5G flow. In prioritized scheduling, the 1G flow has the least priority and therefore sees an increase in its transport delay.

In Fig. 8, we show the maximum observed flow delays as we increase the number of radios in the network. Assuming a fixed number of radios per edge switch, the scaling is achieved by using $q^{2}$ edge switches, and $q$ aggregation switches for a $q$-ary aggregation tree. The core switch capacity, however, is fixed at 200Gbps. Despite quadrupling the number of radios (from 16 to 64), the maximum delay in the network increases not more than 300$ns$ (a less than 10% increase). This is attributed to the maximum delay bound that is dependent only on the height of the tree, and not on the number of radios. Note that this constant increase is not always the case, for example, in daisy-chained radio network in ARGOS [4], the maximum delay grows linearly with the number of radios.

IV Achievable Capacity Under E2E Schedulability

In this section, we characterize the achievable wireless capacity under the constraint of e2e schedulability of baseband transport. We consider a basic implementation of DISTRO with single aggregation switch, $q$ edge switches, and MIMO decoding of spatially-multiplexed baseband samples from $n$ radios. The specifics of synchronization, buffering and decoding are omitted but are assumed to manifest through the requested e2e delay bounds (Sec. II-B).

Assuming everything else is fixed, the ADC quantization determines the flow rate of a radio. It also affects the wireless capacity through quantization noise and thus is the search parameter to maximize capacity while ensuring schedulability.

Fig. 9 shows our system model where samples received by radios over a wireless channel are transported for decoding. Let $\mathbf{x}\in\mathbb{C}^{m\times 1}$ be the signal vector sent by the transmitter, and let $\mathbf{y}\in\mathbb{C}^{n\times 1}$ be the received signal vector, where $m$ is the number of transmit antennas, and $n$ is the number of radios. Let $\mathbf{H}\in\mathbb{C}^{n\times m}$ represent the wireless channel between the transmitter and the radios.

Quantization. Assume radio $i$ has ADC quantization width, $Q_{i}$, that takes an integer value from the set $\mathcal{L}=\{L_{1},\ldots,L_{d}\}$, where $L_{1}$ and $L_{d}$ are the minimum and the maximum quantization widths, respectively. Further, let $\mathbf{Q}=[Q_{1},\ldots,Q_{n}]$, and let $\gamma(Q_{i})$ be the average quantization noise power injected into the baseband samples, where $\gamma(\cdot)$ is the quantization noise function.

Transport. Assume a fixed size $B$ of baseband packets. From Eq. (1) and Eq. (2), the inter-arrival time at radio $i$ is $T_{i}=\frac{B}{2Q_{i}f_{i}}$, which is a function of the radio quantization width, $Q_{i}$.

Model. Assuming a narrow-band channel, the received signal can be expressed as:

[TABLE]

where $E[\mathbf{x}\mathbf{x}^{H}]=\rho\mathbf{I}_{m}$, $\rho$ denotes the transmitted power that is equal across all symbols, $\mathbf{z}\sim\mathcal{C}\mathcal{N}(0,\sigma^{2}\mathbf{I}_{n})$ is the additive complex white gaussian noise, and $\mathbf{z}_{Q}$ represents the quantization noise vector. We assume the effect of quantization manifests through the additive term $\mathbf{z}_{Q}$, which is approximated as a zero-mean complex Gaussian noise with covariance matrix $\mathbf{\Sigma}_{Q}=\text{diag}(\gamma(Q_{i}),\ldots,\gamma(Q_{n}))$. Let $\mathbf{\Sigma}=\sigma^{2}\mathbf{I}_{n}+\mathbf{\Sigma}_{Q}$ denote the equivalent noise covariance matrix.

The ergodic wireless capacity (in b/s/Hz) for a fixed $\mathbf{Q}$, with imperfect channel knowledge at the transmitter, is [23, Eq. 20]:

[TABLE]

where the expectation $\mathrm{E}_{\mathbf{H}}[\cdot]$ is taken over all channel realizations of $\mathbf{H}$.

IV-A Problem Formulation

We want to select the quantization, $\mathbf{Q}$, that ensures e2e schedulability, and also maximizes the wireless capacity. This is expressed through an optimization problem (OP):

[TABLE]

where $D_{i}$ is the transport delay bound of radio $i$. Solving OP is not practical as there exists no e2e schedulability test for Eq. (15). Instead, we solve the problem OP’ for the restricted set of flows used in DISTRO for which schedulability test exists at the edge switches, that is:

[TABLE]

where $d_{i}^{{}^{\prime}}$ is defined as in Theorem 1. Next, we show the following properties for the optimization problem.

Assuming the quantization noise is only dependent on the number of quantization bits, its power generally decreases with increasing resolution (e.g. $\gamma(x)\propto 2^{-2x}$ for uniform-level quantization).

Proposition 1**.**

If $\gamma(\cdot)$ is a monotonically decreasing function, then $R(\mathbf{Q})$ is monotonically increasing in $\mathbf{Q}$.

Proof.

Let $\mathbf{Z}=\mathbf{I}+\rho\mathbf{\Sigma}^{-1}\mathbf{H}\mathbf{H}^{H}$, $\mathbf{\Sigma}^{-1}=\text{diag}(\mu_{1},\ldots,\mu_{n})$, where $\mu_{i}=1/(\sigma^{2}+\gamma(Q_{i}))$ for $i=1,\ldots,n$, and let $\lambda_{1},\ldots,\lambda_{n}$ denote the eigen values of $\mathbf{Z}$ such that $\lambda_{1}\geq\lambda_{2}\geq\ldots\geq\lambda_{n}$. We follow an approach similar to [24, Theorem 10.3] for the proof. For a fixed $i$, we claim that $\lambda_{j}$, for every $j$, is monotonically increasing in $\mu_{i}$. On the contrary, suppose this is not true and there exists an interval of $\mu_{i}$, and $k$, $k\in\{1,\ldots,n\}$, such that $\lambda_{k}$ increases and decreases in that interval. Consequently there exists $\lambda^{{}^{\prime}}$ such that $(\lambda^{{}^{\prime}}-\lambda_{k})$ vanishes for at least two values of $\mu_{i}$. Since $\text{det}(\lambda\mathbf{I}-\mathbf{Z})$ $=\Pi_{j=1}^{n}(\lambda-\lambda_{j})$, then, for $\lambda=\lambda^{{}^{\prime}}$, $\text{det}(\lambda\mathbf{I}-\mathbf{Z})$ vanishes for more than one value of $\mu_{i}$, which is impossible, since $\text{det}(\lambda\mathbf{I}-\mathbf{Z})$ is a linear polynomial in $\mu_{i}$. Therefore if $\gamma(Q_{i})$ strictly decreases with $Q_{i}$, $\mu_{i}$ strictly increases with $Q_{i}$, thus, $\forall j$, $\lambda_{j}$ monotonically increases with $Q_{i}$, for any $i=1,\ldots,n$. Since $\log_{2}\text{det}(\mathbf{Z})=\sum_{j=1}^{n}\log_{2}\lambda_{j}$, and expectation is a monotonic operator, therefore, $R(\mathbf{Q})=\mathrm{E}_{\mathbf{H}}[\log_{2}\text{det}(\mathbf{Z})]$ is monotonically increasing with $Q_{i}$, $\forall i$.     \sqcap$$\sqcup

Proposition 2**.**

For quantization $\mathbf{Q}$, if the set of traffic flows, $(T_{i},d_{i}^{{}^{\prime}}),\forall i\in S_{k}$ is schedulable for all edge switches $k$, where $d_{i}^{{}^{\prime}}$ is as defined in Theorem 1, then the traffic flows $(T_{i}^{{}^{\prime}},d_{i}^{{}^{\prime}}),\forall i\in S_{k}$ are also schedulable for all $k$, for quantization, $\mathbf{Q}^{{}^{\prime}}$, such that $Q_{i}^{{}^{\prime}}\leq Q_{i},\forall i$.

Proof.

Let $\mathcal{K}^{{}^{\prime}}$ be the set of edge switches where radios see the reduction in their quantization width. Consider any $k\in\mathcal{K}^{{}^{\prime}}$: for all $i\in S_{k}$, changing $Q_{i}$ to $Q_{i}^{{}^{\prime}}$ increases the inter-arrival period from $T_{i}$ to $T_{i}^{{}^{\prime}}$. For simplicity, assume the switch implements EDF scheduling and therefore Eq. (19) is true for flows $(T_{i},d_{i}^{{}^{\prime}}),i\in S_{k}$. Since $T_{i}<T_{i}^{{}^{\prime}}$ implies $\lceil{(t-d_{i}^{{}^{\prime}})/T_{i}^{{}^{\prime}}}\rceil^{+}\leq\lceil{(t-d_{i}^{{}^{\prime}})/T_{i}}\rceil^{+}$, $\forall t$, substituting in Eq. (19), it follows that $(T_{i}^{{}^{\prime}},d_{i}^{{}^{\prime}})$, $i=1,\ldots,n$, is also schedulable using Theorem A.1. A similar exercise with Eq. (22) shows that the flows are also schedulable for fixed-priority non-preemptive scheduling given by Theorem A.2.     \sqcap$$\sqcup

IV-B Search Algorithm

The search space of OP*′* is $\mathcal{L}^{n}$, hence, finding the optimal $\mathbf{Q}$ through a brute-force search has an exponential complexity. Note that we cannot relax the integral constraint because of the discreteness of the schedulability. However, from the monotonic dependence on $\mathbf{Q}$ (Propositions 1–2), we can construct a greedy search algorithm.

The idea is to use breadth-first search (BFS) on the enumeration of the search space. Starting from the highest quantization, $[L_{d},\ldots,L_{d}]$, we enumerate the next highest quantizations, $[L_{d-1},L_{d},\ldots,L_{d}]$, $\ldots$, $[L_{d},L_{d},\ldots,L_{d-1}]$, and then enumerate the next highest quantization for each of them, and so on. More generally, let us define the function $enum(\cdot)$, for quantization, $\mathbf{Q}=[L_{k_{1}},L_{k_{2}},\ldots,L_{k_{n}}]$, as follows:

[TABLE]

As we enumerate each possible quantization, we check the flow shcedulability (according to Eq. (17)). If the flows are schedulable, we calculate the corresponding capacity, but do not enumerate the quantization further. Finally, from the calculated capacities, we find quantization $\mathbf{Q}^{\star}$ that achieves the maximum capacity.

Algorithm 1 gives the pseudo-code for implementing our proposed solution. It uses a queue data structure, $U$, for breadth-first traversal. We show that this algorithm finds a solution to OP*′*. Though in the worst case it has the same $O(d^{n})$ complexity, the running time in practice is much less than a brute-force search.

Theorem 2**.**

Algorithm 1 gives the optimal solution to OP′.

Proof.

We show that $\mathbf{Q}^{\star}$ is both schedulable and achieves maximum capacity. Clearly, the schedulability of $\mathbf{Q}^{\star}$ holds from Line 6 in the algorithm. We know from Proposition 1, the capacity from enumeration (in Line 11) always decreases, and hence $C^{\star}$ will always be larger than the capacity of all unenumerated quantizations. Note that from Proposition 2), when a given quantization is schedulable, all its enumerations with smaller quantizations are also schedulable. Also, from Line 7, $C^{\star}$ is the maximum of all enumerated quantizations that are schedulable. Therefore, $C^{\star}$ is the optimal capacity that is schedulable. $\Box$

In summary, DISTRO’s delay characteristics not only provides real-time guarantees but also supports the design of an optimally performing network.

V Related Work

Fat-Tree topology was first introduced as a routing network for parallel computation for networks-on-chip[25]; its formal definition as ”k–ary n–trees” is given in [26]. In the context of datacenter networks, where commodity (off-the-shelf) switches are used, and have a fixed number of ports, fat-tree topologies (or folded clos networks [27]) have been used to build scalable and modular architectures [9, 10]. As these networks are bandwidth bound, they cannot meet real-time requirements. We note that the Fat-Tree variant used in this paper is the simplest kind with no interconnections between the edge and core switches.

A large-scale MU-MIMO was realized using a distributed architecture of servers in [20]. The authors claim that bandwidth needed for baseband transport is not an issue as modern switches support up to 40Gbps links. However, they do not consider the real-time guarantees of transporting baseband samples. ARGOS [4] is another practical multi-antenna setup that suggests daisy-chained radios with Tree-based aggregation. However, no real-time analysis was provided there. C-RAN [1] also proposes a data-center model for baseband processing, which must be carefully designed as these networks are known to have varying traffic and congestion patterns [28].

To eliminate jitter, scheduled Ethernet using global scheduling has been proposed for fronthaul transport in a C-RAN[29]. On the other hand, the authors of [11, 12] studied the compression of baseband signals. They propose quantization schemes for lossy compression of the baseband samples. In [11], the authors propose prioritization of baseband frames, though, the priorities are not based on packet delays and therefore cannot provide e2e guarantees.

Real-time transport of Ethernet packets is a well-known problem in real-time systems literature [16, 7, 8]. While [16] provides a general approach for schedulability, the authors propose the use of flow regulators at each switch, which is difficult to realize as compared to DISTRO’s edge-switch scheduling.

VI Conclusion

This paper provides a design for baseband transport network based on Fat-Tree topology that is scalable to a large number of radios while guaranteeing real-time delivery. We calculate the delay bound of baseband traffic and provide sufficient criteria for e2e schedulability, which is validated via simulations. Certain design restrictions such as symmetric links and equal packet sizes are made but may be relaxed as long as a bound for queuing time at each switch is obtainable.

We also characterize the wireless capacity with the schedulability constraint, and provide an efficient search algorithm to maximize it. In practice, our design and analysis provides a principled approach to the deployment of baseband transport networks that are critical for upcoming wireless system architectures.

VII Appendix

Theorem A.1**.**

Under deadline scheduling, a set of traffic flows $\tau_{i}=(T_{i},d_{i}),i=1,2,\ldots,n$, is schedulable on a link with transmission time $C$, and preemption if and only if [13, Theorem 1]:

[TABLE]

and without preemption if and only if [13, Theorem 6]:

[TABLE]

where $d_{min}=\min\{d_{i}:1\leq i\leq n\}$ and the function $\lceil x\rceil^{+}=max(0,\lceil x\rceil)$.

Theorem A.2**.**

Under fixed-priority scheduling, a set of traffic flows $\tau_{i}=(T_{i},d_{i}),i=1,2,\ldots,n$, and priorities $\pi_{i}$, $i=1,2,\ldots,n$, such that $\pi_{1}\geq\pi_{2}\geq\ldots\geq\pi_{n}$, is schedulable on a link with transmission time $C$, if the function $W_{m}(k,x)$ satisfies:

[TABLE]

where $N_{m}=\min\{k:W_{m}(k,kT_{m})\leq 1\}$, and function $W_{m}(k,x)$, for preemption, is defined as [14]:

[TABLE]

and without preemption, $W_{m}(k,x)$ is defined as [30]:

[TABLE]