Drift-Based Dataset Stability Benchmark

TL;DR

This paper introduces a novel methodology and benchmark workflow for evaluating dataset stability in network traffic classification, addressing concept drift and dataset degradation over time with a focus on improving dataset quality and model robustness.

Contribution

The paper presents a new framework for assessing dataset stability using concept drift detection and feature weights, along with an initial benchmark for network traffic datasets.

Findings

Demonstrated the framework on CESNET-TLS-Year22 dataset.

Identified dataset weak points and stability issues.

Showed how dataset variants can be optimized using the benchmark.

Abstract

Machine learning (ML) represents an efficient and popular approach for network traffic classification. However, network traffic classification is a challenging domain, and trained models may degrade soon after deployment due to the obsolete datasets and quick evolution of computer networks as new or updated protocols appear. Moreover, significant change in the behavior of a traffic type (and, therefore, the underlying features representing the traffic) can produce a large and sudden performance drop of the deployed model, known as a data or concept drift. In most cases, complete retraining is performed, often without further investigation of root causes, as good dataset quality is assumed. However, this is not always the case and further investigation must be performed. This paper proposes a novel methodology to evaluate the stability of datasets and a benchmark workflow that can be used to compare datasets. The proposed framework is based on a concept drift detection method that also uses ML feature weights to boost the detection performance. The benefits of this work are demonstrated on CESNET-TLS-Year22 dataset. We provide the initial dataset stability benchmark that is used to describe dataset stability and weak points to identify the next steps for optimization. Lastly, using the proposed benchmarking methodology, we show the optimization impact on the created dataset variants.