M3S-UPD: Efficient Multi-Stage Self-Supervised Learning for Fine-Grained Encrypted Traffic Classification with Unknown Pattern Discovery

TL;DR

This paper introduces M3S-UPD, a self-supervised learning framework that improves fine-grained encrypted traffic classification and unknown pattern detection without prior knowledge, addressing real-world challenges like data scarcity and concept drift.

Contribution

It presents a novel multi-stage self-supervised approach that unifies classification and detection, enabling effective unknown pattern discovery in encrypted traffic.

Findings

Outperforms existing methods on few-shot classification

Achieves competitive zero-shot unknown traffic detection

Resistant to performance degradation over continuous learning

Abstract

The growing complexity of encrypted network traffic presents dual challenges for modern network management: accurate multiclass classification of known applications and reliable detection of unknown traffic patterns. Although deep learning models show promise in controlled environments, their real-world deployment is hindered by data scarcity, concept drift, and operational constraints. This paper proposes M3S-UPD, a novel Multi-Stage Self-Supervised Unknown-aware Packet Detection framework that synergistically integrates semi-supervised learning with representation analysis. Our approach eliminates artificial segregation between classification and detection tasks through a four-phase iterative process: 1) probabilistic embedding generation, 2) clustering-based structure discovery, 3) distribution-aligned outlier identification, and 4) confidence-aware model updating. Key innovations include a self-supervised unknown detection mechanism that requires neither synthetic samples nor prior knowledge, and a continuous learning architecture that is resistant to performance degradation. Experimental results show that M3S-UPD not only outperforms existing methods on the few-shot encrypted traffic classification task, but also simultaneously achieves competitive performance on the zero-shot unknown traffic discovery task.