Comparative Evaluation of VAE, GAN, and SMOTE for Tor Detection in Encrypted Network Traffic

TL;DR

This paper compares VAE, GAN, and SMOTE for generating synthetic encrypted network traffic to improve intrusion detection, highlighting the strengths and limitations of each method in preserving data characteristics and classifier performance.

Contribution

It provides a comprehensive evaluation of GAI models for synthetic traffic generation, demonstrating their effectiveness in enhancing encrypted traffic detection.

Findings

VAE offers the best privacy-performance balance.

GANs produce high-fidelity data but risk overfitting.

SMOTE improves recall but may lack diversity.

Abstract

Encrypted network traffic poses significant challenges for intrusion detection due to the lack of payload visibility, limited labeled datasets, and high class imbalance between benign and malicious activities. Traditional data augmentation methods struggle to preserve the complex temporal and statistical characteristics of real network traffic. To address these issues, this work explores the use of Generative AI (GAI) models to synthesize realistic and diverse encrypted traffic traces. We evaluate three approaches: Variational Autoencoders (VAE), Generative Adversarial Networks (GAN), and SMOTE (Synthetic Minority Over-sampling Technique), each integrated with a preprocessing pipeline that includes feature selection and class balancing. The UNSW NB-15 dataset is used as the primary benchmark, focusing on Tor traffic as anomalies. We analyze statistical similarity between real and synthetic data, and assess classifier performance using metrics such as Accuracy, F1-score, and AUC-ROC. Results show that VAE-generated data provides the best balance between privacy and performance, while GANs offer higher fidelity but risk overfitting. SMOTE, though simple, enhances recall but may lack diversity. The findings demonstrate that GAI methods can significantly improve encrypted traffic detection when trained with privacy-preserving synthetic data.