Multi-Temporal Analysis and Scaling Relations of 100,000,000,000 Network Packets

TL;DR

This paper introduces an efficient method for analyzing massive network traffic data, revealing new scaling relationships and insights into normal network behavior that can enhance anomaly detection and AI applications.

Contribution

The authors developed a scalable approach for computing streaming network quantities on large datasets, enabling the discovery of previously unobserved scaling laws in network traffic.

Findings

Identification of new scaling relationships in network traffic

Insights into normal background network behavior

Potential applications in anomaly detection and AI feature engineering

Abstract

Our society has never been more dependent on computer networks. Effective utilization of networks requires a detailed understanding of the normal background behaviors of network traffic. Large-scale measurements of networks are computationally challenging. Building on prior work in interactive supercomputing and GraphBLAS hypersparse hierarchical traffic matrices, we have developed an efficient method for computing a wide variety of streaming network quantities on diverse time scales. Applying these methods to 100,000,000,000 anonymized source-destination pairs collected at a network gateway reveals many previously unobserved scaling relationships. These observations provide new insights into normal network background traffic that could be used for anomaly detection, AI feature engineering, and testing theoretical models of streaming networks.