Likelihood-based inference for modelling packet transit from thinned flow summaries

TL;DR

This paper develops a likelihood-based method for analyzing network traffic data that accounts for packet thinning and flow summarization, enabling accurate inference from summarized data.

Contribution

It introduces a novel likelihood-based inference approach that incorporates packet thinning and flow summaries, improving practical traffic modeling accuracy.

Findings

Estimator is consistent under certain conditions.

Derived bounds on data volume needed for accurate inference.

Validated method on real network traffic data.

Abstract

The substantial growth of network traffic speed and volume presents practical challenges to network data analysis. Packet thinning and flow aggregation protocols such as NetFlow reduce the size of datasets by providing structured data summaries, but conversely this impedes statistical inference. Methods which aim to model patterns of traffic propagation typically do not account for the packet thinning and summarisation process into the analysis, and are often simplistic, e.g.~method-of-moments. As a result, they can be of limited practical use. We introduce a likelihood-based analysis which fully incorporates packet thinning and NetFlow summarisation into the analysis. As a result, inferences can be made for models on the level of individual packets while only observing thinned flow summary information. We establish consistency of the resulting maximum likelihood estimator, derive bounds on the volume of traffic which should be observed to achieve required levels of estimator accuracy, and identify an ideal family of models. The robust performance of the estimator is examined through simulated analyses and an application on a publicly available trace dataset containing over 36m packets over a 1 minute period.