Towards Reproducible Network Traffic Analysis

TL;DR

This paper addresses the reproducibility crisis in network traffic analysis by proposing a standardization framework and introducing pcapML, an open source system for encoding metadata into traffic captures, along with benchmarks to track progress.

Contribution

It introduces pcapML, a novel standardization system for network traffic data, and establishes benchmarks to improve reproducibility and comparability of analysis methods.

Findings

Evidence of irreproducibility due to dataset interpretation differences

pcapML enables consistent encoding of metadata in traffic captures

Benchmark platform tracks progress of analysis techniques

Abstract

Analysis techniques are critical for gaining insight into network traffic given both the higher proportion of encrypted traffic and increasing data rates. Unfortunately, the domain of network traffic analysis suffers from a lack of standardization, leading to incomparable results and barriers to reproducibility. Unlike other disciplines, no standard dataset format exists, forcing researchers and practitioners to create bespoke analysis pipelines for each individual task. Without standardization researchers cannot compare "apples-to-apples", preventing us from knowing with certainty if a new technique represents a methodological advancement or if it simply benefits from a different interpretation of a given dataset. In this work, we examine irreproducibility that arises from the lack of standardization in network traffic analysis. First, we study the literature, highlighting evidence of irreproducible research based on different interpretations of popular public datasets. Next, we investigate the underlying issues that have lead to the status quo and prevent reproducible research. Third, we outline the standardization requirements that any solution aiming to fix reproducibility issues must address. We then introduce pcapML, an open source system which increases reproducibility of network traffic analysis research by enabling metadata information to be directly encoded into raw traffic captures in a generic manner. Finally, we use the standardization pcapML provides to create the pcapML benchmarks, an open source leaderboard website and repository built to track the progress of network traffic analysis methods.