New Phenomena in Large-Scale Internet Traffic

TL;DR

This paper analyzes large-scale Internet traffic data, revealing the significance of leaf nodes and links, and introduces a modified Zipf-Mandelbrot model that accurately describes traffic patterns across diverse network streams.

Contribution

It presents the largest publicly available Internet traffic datasets and introduces a new statistical model that captures traffic characteristics across various network topologies.

Findings

Unveiled the importance of leaf nodes and isolated links in Internet traffic.

A modified Zipf-Mandelbrot distribution accurately models source/destination statistics.

Model parameters correlate with underlying network topologies.

Abstract

The Internet is transforming our society, necessitating a quantitative understanding of Internet traffic. Our team collects and curates the largest publicly available Internet traffic data sets. An analysis of 50 billion packets using 10,000 processors in the MIT SuperCloud reveals a new phenomenon: the importance of otherwise unseen leaf nodes and isolated links in Internet traffic. Our analysis further shows that a two-parameter modified Zipf-Mandelbrot distribution accurately describes a wide variety of source/destination statistics on moving sample windows ranging from 100{,}000 to 100{,}000{,}000 packets over collections that span years and continents. The measured model parameters distinguish different network streams, and the model leaf parameter strongly correlates with the fraction of the traffic in different underlying network topologies.