Systematic Evaluation of Synthetic Data Augmentation for Multi-class NetFlow Traffic

TL;DR

This study systematically compares classical and generative data augmentation methods for multi-class network intrusion detection, revealing that resampling techniques do not consistently improve classifier performance across various datasets.

Contribution

It provides a comprehensive framework for evaluating data balancing methods in NIDS, highlighting the limited and inconsistent benefits of resampling techniques.

Findings

Resampling methods do not reliably enhance classification performance.

Some instances show performance improvements, but overall results are inconsistent.

No resampling technique consistently outperforms others across classifiers and datasets.

Abstract

The detection of cyber-attacks in computer networks is a crucial and ongoing research challenge. Machine learning-based attack classification offers a promising solution, as these models can be continuously updated with new data, enhancing the effectiveness of network intrusion detection systems (NIDS). Unlike binary classification models that simply indicate the presence of an attack, multi-class models can identify specific types of attacks, allowing for more targeted and effective incident responses. However, a significant drawback of these classification models is their sensitivity to imbalanced training data. Recent advances suggest that generative models can assist in data augmentation, claiming to offer superior solutions for imbalanced datasets. Classical balancing methods, although less novel, also provide potential remedies for this issue. Despite these claims, a comprehensive comparison of these methods within the NIDS domain is lacking. Most existing studies focus narrowly on individual methods, making it difficult to compare results due to varying experimental setups. To close this gap, we designed a systematic framework to compare classical and generative resampling methods for class balancing across multiple popular classification models in the NIDS domain, evaluated on several NIDS benchmark datasets. Our experiments indicate that resampling methods for balancing training data do not reliably improve classification performance. Although some instances show performance improvements, the majority of results indicate decreased performance, with no consistent trend in favor of a specific resampling technique enhancing a particular classifier.