# Hypersparse Neural Network Analysis of Large-Scale Internet Traffic

## TL;DR

This paper introduces a hypersparse neural network method to analyze large-scale Internet traffic, revealing new traffic phenomena and modeling traffic distributions with high accuracy across diverse network data.

## Contribution

The study presents a novel hypersparse neural network approach for analyzing massive Internet traffic data, uncovering previously unseen network features and accurately modeling traffic distributions.

## Key findings

- Identification of the importance of leaf nodes and isolated links in traffic.
- A two-parameter Zipf-Mandelbrot distribution models traffic statistics effectively.
- Model parameters correlate with network topologies.

## Abstract

The Internet is transforming our society, necessitating a quantitative understanding of Internet traffic. Our team collects and curates the largest publicly available Internet traffic data containing 50 billion packets. Utilizing a novel hypersparse neural network analysis of "video" streams of this traffic using 10,000 processors in the MIT SuperCloud reveals a new phenomena: the importance of otherwise unseen leaf nodes and isolated links in Internet traffic. Our neural network approach further shows that a two-parameter modified Zipf-Mandelbrot distribution accurately describes a wide variety of source/destination statistics on moving sample windows ranging from 100,000 to 100,000,000 packets over collections that span years and continents. The inferred model parameters distinguish different network streams and the model leaf parameter strongly correlates with the fraction of the traffic in different underlying network topologies. The hypersparse neural network pipeline is highly adaptable and different network statistics and training models can be incorporated with simple changes to the image filter functions.