Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic

TL;DR

This paper introduces RAPIER, a robust framework for detecting encrypted malicious network traffic in low-quality datasets, utilizing data distribution insights and label correction to significantly improve detection performance.

Contribution

The paper presents RAPIER, a novel system that leverages distribution-based data augmentation and label noise correction to enhance encrypted malicious traffic detection in low-quality datasets.

Findings

RAPIER achieves high F1 scores on public datasets with noisy data.

It significantly outperforms existing methods, with improvements over 200%.

Effective in real-world enterprise traffic detection.

Abstract

Machine learning (ML) is promising in accurately detecting malicious flows in encrypted network traffic; however, it is challenging to collect a training dataset that contains a sufficient amount of encrypted malicious data with correct labels. When ML models are trained with low-quality training data, they suffer degraded performance. In this paper, we aim at addressing a real-world low-quality training dataset problem, namely, detecting encrypted malicious traffic generated by continuously evolving malware. We develop RAPIER that fully utilizes different distributions of normal and malicious traffic data in the feature space, where normal data is tightly distributed in a certain area and the malicious data is scattered over the entire feature space to augment training data for model training. RAPIER includes two pre-processing modules to convert traffic into feature vectors and correct label noises. We evaluate our system on two public datasets and one combined dataset. With 1000 samples and 45% noises from each dataset, our system achieves the F1 scores of 0.770, 0.776, and 0.855, respectively, achieving average improvements of 352.6%, 284.3%, and 214.9% over the existing methods, respectively. Furthermore, We evaluate RAPIER with a real-world dataset obtained from a security enterprise. RAPIER effectively achieves encrypted malicious traffic detection with the best F1 score of 0.773 and improves the F1 score of existing methods by an average of 272.5%.