Hypersparse Network Flow Analysis of Packets with GraphBLAS

TL;DR

This paper introduces a novel GraphBLAS-based hypersparse matrix method for compressing and analyzing large-scale network flow data, enabling efficient temporal aggregation, anomaly detection, and background modeling at unprecedented scale.

Contribution

It presents a new hypersparse matrix compression and resampling technique using GraphBLAS that preserves privacy and allows detailed temporal analysis of network flows.

Findings

Achieved compression below 0.1 bits per packet.

Performed analysis on over a million packets per second.

Scalability demonstrated on the MIT SuperCloud with hundreds of sites.

Abstract

Internet analysis is a major challenge due to the volume and rate of network traffic. In lieu of analyzing traffic as raw packets, network analysts often rely on compressed network flows (netflows) that contain the start time, stop time, source, destination, and number of packets in each direction. However, many traffic analyses benefit from temporal aggregation of multiple simultaneous netflows, which can be computationally challenging. To alleviate this concern, a novel netflow compression and resampling method has been developed leveraging GraphBLAS hyperspace traffic matrices that preserve anonymization while enabling subrange analysis. Standard multitemporal spatial analyses are then performed on each subrange to generate detailed statistical aggregates of the source packets, source fan-out, unique links, destination fan-in, and destination packets of each subrange which can then be used for background modeling and anomaly detection. A simple file format based on GraphBLAS sparse matrices is developed for storing these statistical aggregates. This method is scale tested on the MIT SuperCloud using a 50 trillion packet netflow corpus from several hundred sites collected over several months. The resulting compression achieved is significant (<0.1 bit per packet) enabling extremely large netflow analyses to be stored and transported. The single node parallel performance is analyzed in terms of both processors and threads showing that a single node can perform hundreds of simultaneous analyses at over a million packets/sec (roughly equivalent to a 10 Gigabit link).