Measuring the Complexity of Packet Traces

TL;DR

This paper introduces a systematic method to quantify and compare the complexity of packet traces using entropy, providing insights into traffic patterns and enabling the creation of realistic traffic generators.

Contribution

It presents a novel approach to measure trace complexity through entropy analysis and develops a traffic generator that replicates the complexity of real-world traces.

Findings

Trace complexity varies across different applications.

The proposed model accurately reproduces the complexity of real traces.

Entropy-based metrics reveal structural differences in packet traces.

Abstract

This paper studies the structure of several real-world traces (including Facebook, High-Performance Computing, Machine Learning, and simulation generated traces) and presents a systematic approach to quantify and compare the structure of packet traces based on the entropy contained in the trace file. Insights into the structure of packet traces can lead to improved network algorithms that are optimized toward specific traffic patterns. We then present a methodology to quantify the temporal and non-temporal components of entropy contained in a packet trace, called the trace complexity, using randomization and compression. We show that trace complexity provides unique insights into the characteristics of various applications and argue that there is a need for traffic generation models that preserve the intrinsic structure of empirically measured application traces. We then propose a traffic generator model that is able to produce a synthetic trace that matches the complexity level of its corresponding real-world trace.

Tables

Table 1. Table 1: Traces used in the paper.

Type	# traces	Sources	Dests	Avg.# of entries
Machine Learning [15]	1	4	4	1869
Facebook (IP) [19]	3	174K	156K	31M
Facebook (Rack) [19]	3	282	15K	31M
HPC [20]	3	1024	1024	11M
pFabric [14]	3	144	144	30M
Reference Points (own)	4	16	16	10M

Equations

$$\Psi (\sigma )=\frac{C(\sigma )}{C(\mathcal{U}(\sigma ))}.$$

$$y=H(M)/(2\log n)$$

$$x=\frac{H(p,1-p)+(1-p)H(M)}{H(M)}$$

$$\underset{\begin{array}{c}t\to \infty \end{array}}{\lim }\Psi (\sigma )=\frac{C(\sigma )}{C(\mathcal{U}(\sigma ))}=\frac{H(\mathcal{Z})}{2\log n}$$

$$\underset{\begin{array}{c}t\to \infty \end{array}}{\lim }T(\sigma )=\frac{C(\sigma )}{C(\Gamma (\sigma ))}=\frac{H(\mathcal{Z})}{H(S,D)}$$

$$\underset{\begin{array}{c}t\to \infty \end{array}}{\lim }NT(\sigma )=\frac{C(\Gamma (\sigma )))}{C(\mathcal{U}(\sigma ))}=\frac{H(S,D)}{2\log n}$$

Full text

1 Introduction

Packet traces collected from networking applications, such as data center traffic, tend not to be completely random and have been observed to feature structure: data center traffic matrices are sparse and skewed [1, 2], exhibit locality [3], and are bursty [4, 5]. Motivated by the existence of such structure, the networking community is currently putting much effort into designing algorithms to optimize different network layers toward such structure towards self-driving and demand-aware networks [6, 7, 8], learning-based traffic engineering [9] and video streaming [10], as well as reconfigurable optical networks [11, 2, 12]. For instance, many network optimizations exploit the presence of elephant flows [13, 14].

However, the structure available in different applications can differ significantly, and a unified approach to measure the structure in traffic traces is missing. Better quantification of a trace structure will lead to better network optimization and to the understanding of the available improvement, if possible, in current solutions. Moreover, one of the critical factors in evaluating new proposals is their traffic workload. Ideally, the traffic workload should contain the same structure as real-world traces but often is overlooked due to the lack of a traffic generation model that can replicate real-world traces while preserving their temporal and non-temporal structure.

For instance, consider a trace file including the communication pattern of a Machine Learning (ML) application executing a popular convolutional neural network training job on four GPUs. This workload was obtained from authors of  [15]. Figure 1(a) is a visualization of the trace file where each entry in the trace is represented by a unique color corresponding to its ¡source, destination¿ GPU pair. This visualization shows the temporal structure in the trace file, as colors appear consecutive and follow some pattern. In contrast, Fig. 1(b) shows the same trace file but the entries in the file are shuffled to remove the temporal structure in Fig. 1(a). The traffic matrix (TM) in Fig. 1(c) shows a skewed heatmap indicating that some GPU pairs communicate more frequently than others. Note that even thought the temporal structures in (a) and (b) are different, they both have the same TM shown in (c). In other words, the TM is able to capture the non-temporal structure in the trace files but not the temporal one. For comparison, let us now consider two synthetic traces shown in Fig. 1(d) and (e). Trace (d) is generated uniformly and random and has the lowest temporal structure compared to (a), while trace (e) is sorted based on ¡source, destination¿ key and hence has the highest temporal structure. Similarly, Fig. 1(f) captures the non-temporal structure in (d) and (e) but not the temporal one.

But how can we measure the structure of a trace file? And how should we generate synthetic traces while preserving the structure of empirical traces? This paper aims at providing initial steps to these questions. In particular, we quantify the amount of temporal and non-temporal structure in traffic traces using the information theoretic measure of entropy [16] in the trace. Since the term entropy is define for random variables, as opposed to a sequence of individual communication requests in a packet trace, we use a more general term called complexity [17] to quantify the structure in a packet trace and call it trace complexity. Moreover, we provide a traffic generation model to generate synthetic traces that match the structure of a given trace. Intuitively, a packet trace with high structure has low entropy and low complexity: it contains little information, and the sequence behavior is more predictable [18]. Our approach allows us to chart, what we call, a complexity map of individual traffic traces: to map each traffic trace to a two-dimensional graph indicating the amount of temporal and non-temporal information that is present in a trace.

The main contributions of this paper are as follows. First, we present an information theoretic perspective to systematically separate the temporal and non-temporal structures available in a traffic trace (§2). Second, we compare the complexity of 17 trace files, shown in Table 1, including production-level traces (Facebook [19] and High Performance Computing [20]), NS2 simulation-based traces (pFabric [14]), Machine Learning workload (SiP-ML [15]), as well as our own generated traces for reference points (for uniform, skewed, and bursty traffic patterns). Our results indicate that production-level traces have high temporal complexity while NS2 generated traces have high non-temporal structure. Third, we show that our methodology can be used to slice the complexity into rack-level versus IP-level, as well as source versus destination structures (§3). Finally, we present a simple yet powerful model to generate traffic traces that match the complexity of production-level traces (§4).

2 Defining Trace Complexity

This section describes our methodology to quantify the inherent structure in packet traces. Given a packet trace, $\sigma$, we define its trace complexity as the ratio of its entropy over that of a random trace, $\mathcal{U}(\sigma)$. Intuitively, a random trace does not compress as well as a more structured trace. At the heart of our methodology lie two main concepts: ($i$) Randomization: we systematically randomize different slices of a traffic trace to profile their contributions to the trace complexity; ($ii$) Compression: we then measure the complexity of the trace and its randomized variants by compressing the trace files. The size of the compressed trace file is then taken as the measure of trace complexity. Next, we explain these two concepts more formally.

Eliminate Structure by Randomization. A traffic trace file $\sigma$ consists of an ordered list of entries $\sigma_{1},\sigma_{2},\dots,\sigma_{t}$, where each entry $\sigma_{i}=(s_{i},d_{i})$ is a ¡source, destination¿ pair arriving at time $i$. In this work, we ignore other fields in a packet header (such as packet size and port number) and focus on the order of entries in the trace file, to capture temporal complexity, and source/destination pairs, to capture non-temporal complexity, of the trace. We find that there is enough information in our methodology to capture the differences in temporal and non-temporal complexities of real traces. For instance, §3.1 shows that Facebook’s Hadoop trace [19] has about 50% less temporal structure compared to pFabric’s trace [14].

Trace Complexity. We now define the trace complexity, $\Psi(\sigma)$, as the ratio between the complexity of the original trace, $\sigma$, to the expected complexity of its randomized counterpart, $\mathcal{U}(\sigma)$, where each of its entries are chosen uniformly at random from the set of IDs in $\sigma$:

[TABLE]

As we will discuss later in this section, $C(\cdot)$ represents the size of the compressed trace file and the more structure a trace has, the better it can be compressed. Hence, $\Psi(\sigma)\in[0,1]$ since $C(\sigma)\leq C(\mathcal{U}(\sigma))$.

Temporal Trace Complexity. The temporal structure of a trace is a reflection of the burstiness in the traffic pattern. Prior work has measured the degree of burstiness in a trace as a sequence data packets with inter-arrival time less or equal to 1 millisecond [21, 22, 5]. Instead, we capture the temporal structure in a trace, $\sigma$, by systematically randomizing the original trace to eliminate all temporal relations in the trace and obtain a new trace file ${\Gamma}(\sigma)$. Intuitively, ${\Gamma}(\sigma)$ is a trace where each communication request is chosen independently at random from the previous ones. Formally, let ${\Gamma}(\sigma)$ be a temporal transformation: a transformation that performs a uniform random permutation of the rows of $\sigma$, eliminating any time dependency between rows, therefore $C(\sigma)\leq C({\Gamma}(\sigma))$. To measure how much temporal complexity is contained in $\sigma$, we therefore normalized it by the complexity of its temporal transformation, ${\Gamma}(\sigma)$. Hence, the normalized temporal trace complexity is defined as $T(\sigma)=\frac{C(\sigma)}{C({\Gamma}(\sigma))}\in[0,1]$.

Non-Temporal Trace Complexity. Note that, non-temporal structure is unaffected by the transformation ${\Gamma}(\sigma)$ because correlations such as requests frequency, source destination dependency are conserved, while only the order of the elements is changed such that it is uniformly random. Therefore, all the remaining complexity after the elimination of temporal complexity is non-temporal. This can be formally defined using our methodology by normalizing ${\Gamma}(\sigma)$ with $\mathcal{U}(\sigma)$ which has maximum complexity and no structure: $NT(\sigma)=\frac{C({\Gamma}(\sigma)))}{C(\mathcal{U}(\sigma))}\in[0,1]$.

Theoretical Properties. It directly follows from our definition that the measure of trace complexity $\Psi(\sigma)$ defined in Eq. 1 is the multiplication of the temporal and non-temporal complexity ratios. Formally, $\Psi(\sigma)=T(\sigma)\times NT(\sigma)$. An important feature of our methodology is that is enables comparing traces of different sizes and domains. Section 4.2 describes the relationship of our metric to the entropy rate of a trace when $\sigma$ is generated by a stationary stochastic process.

Compression-based Complexity. Our methodology to measure the empirical entropy of a trace file relies on the principle of data compression. The better we can compress a traffic trace, $\sigma$, the lower must be its entropy rate and hence its complexity. We assume a compression function or algorithm $C$ is applied to $\sigma$ and the complexity of $\sigma$, $C(\sigma)$, is the size of the compressed trace file. In this work, we use the 7zip compressor with Lempel-Ziv-Markov chain compression (LZMA) [23]; other compression techniques such as DEFLATE [24] can also be used.

3 Quantifying the Complexity of Real-world Traces

In this section, we first propose a graphical representation, called the complexity map, to quantify and compare the temporal and non-temporal complexities of different traces on a 2-dimensional plane (§3.1). We then use the complexity map to draw three main takeaways of our analysis on real-world traces (§3.2).

Setup and Dataset. As shown in Table 1, our dataset consists of 17 trace files in five categories: ($i$) trace files from three Facebook (FB) [19] datacenters: hadoop (HAD), web (WEB), and database (DB) including IP and rack-level traces; ($ii$) MPI traces of three exascale applications in high performance computing (HPC) clusters [20]: CNS, MultiGrid, and NeckBone; ($iii$) pFabric [14] packet traces that we generated by running the NS2 simulation script obtained from the authors of the paper; ($iv$) a machine learning (ML) trace we obtained from [15] that measures the communication pattern between four GPUs running VGG19, a popular convolutional neural network training job; and $(v)$ four reference traces that we synthetically generate to represent bursty, skewed, busty & skewed, and uniform traces. To avoid result distortion due to non-random ID selection in production traces, we uniformly hash all the source/destination IDs to the same length and domain.

3.1 The Complexity Map

To compare the complexity of traces in our dataset, we place them on a complexity map where the X and Y axes represent the temporal and non-temporal complexity dimensions. Fig. 2 shows the complexity map of seven real traces and four reference points where each circle indicates a trace and the area of the circles corresponds to the overall complexity of $\sigma$, $\Psi(\sigma)$, which is calculated by multiplying the temporal and non-temporal complexities of $\sigma$, as described in §2.

Before diving into our main takeaways, we describe the four reference points on the complexity map. These points indicate the theoretical complexity of four hypothetical synthetic traces labeled as Uniform, Skewed, Bursty , and Skewed & Bursty. The Uniform trace is located at (1,1), indicating that it has the highest possible complexity and, hence, no structure. This means that the trace is a uniformly chosen random sequence. The Skewed trace is located at (1,0.4), which indicates that it has high temporal complexity and low non-temporal complexity. This is a result of requests in the sequence that are distributed iid (and hence with high temporal complexity), but which arrive from a skewed distribution with low entropy (and hence have low non-temporal complexity). In contrast, the Bursty trace, located at (0.4,1), has low temporal complexity and high non-temporal complexity. This is the case when the next source-destination pair is selected uniformly at random, i.e., with high non-temporal complexity, but then repeated for some time (i.e., modeling a burst), creating temporal patterns and lower temporal complexity. Lastly, the Skewed & Bursty trace, located at (0.4, 0.4), has the lowest complexity in the current map and has both temporal and non-temporal structure. Requests are both temporally dependent (i.e., with repetitions) and new requests arrive from a skewed distribution. All four traces can be generated using a Markovian model which we describe in more details later in Section 4. Fig. 3 shows the traffic matrix of three of the traces in Fig. 2: CNS, MultiGrid, and the uniform reference point. The observation that MultiGrid has less non-temporal complexity than CNS is captured by the differences in their traffic matrices shown in Fig. 3. In particular, we can observe that Fig. 3(a) has less structure than 3(b), hence CNS has higher non-temporal complexity than MultiGrid. In contrast, Fig. 3(c) shows no structure and hence it has the highest complexity in the complexity map in Fig 2.

3.2 Takeaways

In this section, we apply the complexity map to different traces and discuss the main takeaways with respect to their complexities and the differences between them.

Applications have different complexity measures. The complexity map highlights the different characteristics and structures available in different applications, confirming observations such as [19] conducted on Facebook’s datacenters. Recall Fig. 2: pFabric and ML traces feature a higher non-temporal complexity than MultiGrid and all Facebook (DB, HAD, WEB) traces, but pFabric has a lower temporal complexity than all the other traces. Interestingly, Facebook traces have the highest temporal complexity. We suspect this is because of the 30,000 to 1 sampling of Facebook traces that destroys the temporal structure resulting in a high temporal complexity. This indicates that different applications may be identified by their specific complexity characteristic, and may provide different opportunities for optimization. To obtain a more detailed understanding, let us zoom in to the pFabric and HPC traces. Fig. 4(a) shows that the complexity of pFabric also depends on the load (here, 10%, 50%, and 80% loads are shown): at lower loads, fewer flows are competing, and hence flows mix to a lesser extent, naturally resulting in a lower temporal complexity. While this is expected, it validates our methodology and shows that compression can capture this behavior. The non-temporal complexity of different HPC traces (CNS, MultiGrid, and NeckBone) depends on the specific application, but is generally lower than that of pFabric, validating that compression can capture the non-temporal structure, as expected.

Aggregation-level matters. The Facebook traces provide an opportunity for additional insights, as they allow us to study not only IP-to-IP traces but also rack-to-rack traffic. Fig. 4 (b) shows the complexities of three different Facebook clusters, HAD, WEB, and DB. First, we see that some applications (e.g., Hadoop) feature much more structure than others (e.g., DB). However, we also observe the impact of aggregation: at the rack level, we see a higher temporal complexity: communication becomes more random. Moreover, WEB and DB have a slightly lower non-temporal complexity on the rack level, an indication that this traffic has a high structure and placement in datacenters is subject to optimization. In case of HAD, the rack-level complexity is higher than the IP-level complexity, in both dimensions: as we move higher to the topology, communication becomes more uniform. It is important to note that the Facebook traces are sampled from real traffic, but only from a part of the racks in the datacenter [19]. As a result, this trace is not a perfect representation of the entire network, especially in terms of its source distribution. To get more accurate results, without introducing under-estimation in non-temporal complexity, we modified the uniform transformation $\mathcal{U}(\sigma)$ such that it will generate the source and destination columns individually, only from the set of IDs found in the its respective columns. Also, we note that while sampling influences the measured absolute values, it does not affect the relative conclusions, e.g., regarding the relatively higher complexity observed on the rack-level.

The complexity of sources and destinations is unique to each trace. So far, we have focused on communication pairs, however, our methodology also allows us to investigate the complexities introduced by sources and destinations separately. Indeed, traffic matrices may be asymmetric in that while communication sources are more skewed in the traffic trace, communication destinations are uniform, and vice versa. Accordingly, Fig. 4 (c) depicts the complexity map for different traces separated by their source and destinations. In case of pFabric traces, sources and destinations behave similarly, which is expected, given that they are sampled uniformly at random. In the case of HPC, the non-temporal complexity is high, namely, the work seems to be uniformly divided among all CPUs. See also the marginal source and destination distributions in Fig. 3(a). Interestingly, the sources and destinations individually contain temporal structure (where the source has lower complexity), which may be an indication that operations proceed in rounds, e.g., where a node (CPU) first sends several requests and then receives answers asynchronously. The IP traces for WEB application reveal that the sources have less temporal complexity than the destinations, which may be explained by the star-like communication patterns of a web server; the lower non-temporal complexity indicates a more skewed popularity distribution of web servers (compared to cache destinations which are load-balanced). The high non-temporal complexity of Hadoop on the rack-level shows the rather equal distribution, however, temporal structure may be leveraged due to consecutive communications. The low non-temporal complexity of destinations is a result of the outband sampling of the FB trace where the number of sources is much smaller than the number of destinations.

4 Traffic Generation Model

One interesting implication of our methodology is that it naturally lends itself as the basis for synthetic traffic generators. Furthermore, it allows us to provide formal guarantees on the complexity of stochastic processes. In the following, we discuss these two aspects.

4.1 From Analysis to Synthesis

We next tackle the question of how to synthesize traffic workloads of a particular temporal and non-temporal complexity. Given the limited amount of publicly available communication traces, such a model can be particularly useful to generate synthetic benchmarks, allowing researchers to compare their algorithms in different settings (e.g., for longer communication traces). Hence, in the following, we propose an approach which allows to efficiently generate traces with formal guarantees on their expected complexity for any specific point on the complexity map. It is important to note that for all points on the map, there could be many traces (and models) whose complexity maps to this point. Our model provides one such solution.

To derive formal guarantees, we propose a simple Markovian model which is a stationary random process with a well-defined entropy rate [25]. The model has two components: temporal and non-temporal. The non-temporal component is a joint probability traffic matrix, $M$, which can be computed from a given trace $\sigma$, similar to Fig. 1(c), or represent a known distribution (e.g. Zipf) where the entropy depends on the distribution’s parameters. The temporal component is a repeating probability $p$. To generate a trace, we start by sampling the first pair from $M$, and then at each step we add a pair to the trace, with probability $p$ we repeat the last pair and with probability $1-p$ we sample a new pair from $M$. More formally, to emulate a point $(x,y)$ on the complexity map we set $H(M)=y\cdot 2\log n$ where $H(M)$ is the joint entropy of $M$. It can be shown that $M$ is the stationary distribution of the chain and the non-temporal complexity of the model is

[TABLE]

The temporal complexity of the model is

[TABLE]

where $p$ can be computed analytically given $x$. Therefore, we can produce traces with similar complexities on the complexity map.

Figure 5 presents the quality of the above traffic generation model in producing syntactic traces for seven example points in the complexity map. First we used the model to reproduce the four hypothetical points from Fig. 2: Uniform, Skewed, Bursty, and Skewed & Bursty. As a skewed distribution we used Zipf distribution with an exponent parameter of $\frac{2}{3}$ which leads to a normalized entropy of $0.4$ for a trace that is based on 16 IDs and 256 unique source-destination pairs. For the other three traces: ML, MultiGrid, and WEB, we used the joint probability matrix of each trace. By applying the compression methodology to the synthetic traces generated using our model, we find that our empirical traces are relatively close to the original trace files in the complexity map. However, the quality of these results is practically bounded by the selection of the compressing method and the ability of the compressor to capture precisely the entropy rate. Moreover, currently we do not reproduce exact packet arrival times, rather we take the order of packets as a proxy for temporal structure. In future work, we plan to add packet arrival times and flow-level information to the complexity analysis and our traffic generation model.

4.2 Supporting Formal Analysis

Our methodology can also provide a framework for formal analysis. In the following, assume that $\sigma$ is a trace generated by a stationary stochastic process [25]. Then, for long sequences and using an optimal compression algorithm (such as the Lampel-Ziv [26]), we will achieve the compression limit defined by Shannon [25, 27, 28]: the entropy rate of the process. From this, the normalized complexities of $\sigma$ can be proved analytically.

Let $\mathcal{Z}=\{Z_{i}\}$ be a stationary stochastic process that generates $\sigma$ where $Z_{i}=\{S_{i},D_{i}\}$ are time-indexed random variables, and $S_{i}\in S$ and $D_{i}\in D$ are a random source and a random destination at time $i$, respectively. Since $\mathcal{Z}$ is stationary, let $\pi$ denote the stationary distribution and note that $\pi$ is a joint distribution over $S$ and $D$. With a slight abuse of notation, let $S$ and $D$ also denote the random variable of $\pi$. Note that $S$ and $D$ may be defined over different domains (IDs) and may be dependent. Also $Z_{i}$ may be dependent on a past element in $\mathcal{Z}$.

A basic measure for the complexity of $\mathcal{Z}$ is based on Shannon Entropy and known as the entropy rate [25], $H(\mathcal{Z})$. The entropy rate of a stochastic process captures the expected number of bits per symbol (i.e., source or destination IDs) that are both necessary and sufficient to describe $\sigma$; or, alternatively, the expected amount of uncertainty in the next symbol given past symbols in the sequence. The smaller the entropy rate, the less complex is the sequence (it requires fewer bits to describe/compress it).

We can use the previous definitions of normalized trace complexity of $\sigma$ and formally relate them to entropy rates when $\sigma$ is generated by a stationary stochastic process.

Theorem 1** (Trace Complexity Ratios of a Stationary Process).**

Consider an indexed stationary stochastic trace process $\mathcal{Z}=\{Z_{t}\}$ to generate $\sigma$ where $Z_{t}=\{S_{t},D_{t}\}$ and $n=\lvert S\cup D\rvert$. If an optimal compression algorithm $C$ is used, then:

The trace complexity ratio is

[TABLE] 2. 2.

The temporal complexity ratio is:

[TABLE] 3. 3.

The non-temporal complexity ratio is:

[TABLE]

5 Related Work

Information-theoretic approaches and compression methodologies have already been proven successful in capturing entropy in other domains such as email [29], or comment [30] spam filtering, or estimating neural discharges [31]. The study of traffic patterns and the design of models is an evergreen topic of high relevance in the networking literature, and examples where measurement studies spurred much research into traffic modeling dates back to the 1990s [32, 33, 34]. Since then, a large number of methodologies have been developed [35, 36, 37], based on temporal statistics [38, 39, 40], spatial statistics [41, 42], and physical [43] and information-theoretic [44, 45] models. In contrast to prior work, we are primarily interested in the communication pattern itself, rather than in the volume or headers of the exchanged data. While our work builds upon many significant results developed over the last decades [17], we are not aware of any work which allows to systematically differentiate between temporal and non-temporal components of traffic traces.

6 Conclusion

The specific characteristics of traffic workloads have important implications for emerging network fabrics and algorithms [19]. This paper takes the first steps at understanding the complexity of traffic traces. In addition to temporal and non-temporal complexity measures, our entropy-based approach can be used to investigate other dimensions of the trace structure, e.g., regarding source-destination dependencies, or to explore structure in transmission times and packet headers. More generally, while trace structure indicates potential for optimizations, it remains to develop algorithms which exploit this structure to improve network performance and/or utilization.