When Simple Model Just Works: Is Network Traffic Classification in Crisis?

TL;DR

This paper reveals that a simple k-NN baseline can match or outperform complex neural networks in network traffic classification due to dataset redundancies, and it discusses how current evaluation practices may be misleading.

Contribution

The study systematically evaluates a simple k-NN baseline across multiple datasets, uncovers dataset redundancies affecting performance estimates, and proposes new evaluation directions for the field.

Findings

Redundant samples are prevalent in traffic datasets, affecting performance estimates.

Simple k-NN can outperform complex models due to dataset redundancies.

Current evaluation practices may overestimate model accuracy.

Abstract

Machine learning has been applied to network traffic classification (TC) for over two decades. While early efforts used shallow models, the latter 2010s saw a shift toward complex neural networks, often reporting near-perfect accuracy. However, it was recently revealed that a simple k-NN baseline using packet sequences metadata (sizes, times, and directions) can be on par or even outperform more complex methods. In this paper, we investigate this phenomenon further and evaluate this baseline across 12 datasets and 15 TC tasks, and investigate why it performs so well. Our analysis shows that most datasets contain over 50% redundant samples (identical packet sequences), which frequently appear in both training and test sets due to common splitting practices. This redundancy can lead to overestimated model performance and reduce the theoretical maximum accuracy when identical flows have conflicting labels. Given its distinct characteristics, we further argue that standard machine learning practices adapted from domains like NLP or computer vision may be ill-suited for TC. Finally, we propose new directions for task formulation and evaluation to address these challenges and help realign the field.