SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection

TL;DR

This paper critically evaluates the practical effectiveness of machine learning methods for network intrusion detection by proposing a pragmatic assessment approach that considers real-world deployment factors and extensive empirical testing.

Contribution

It introduces the concept of pragmatic assessment for ML in NID and provides a large-scale, reproducible evaluation of existing methods across diverse configurations and hardware.

Findings

Most ML methods lack reliable estimates of their real-world value for NID.

Hardware and configuration significantly impact ML performance in NID.

Practitioners find current evaluation methods insufficient for deployment decisions.

Abstract

Machine Learning (ML) has become a valuable asset to solve many real-world tasks. For Network Intrusion Detection (NID), however, scientific advances in ML are still seen with skepticism by practitioners. This disconnection is due to the intrinsically limited scope of research papers, many of which primarily aim to demonstrate new methods ``outperforming'' prior work -- oftentimes overlooking the practical implications for deploying the proposed solutions in real systems. Unfortunately, the value of ML for NID depends on a plethora of factors, such as hardware, that are often neglected in scientific literature. This paper aims to reduce the practitioners' skepticism towards ML for NID by "changing" the evaluation methodology adopted in research. After elucidating which "factors" influence the operational deployment of ML in NID, we propose the notion of "pragmatic assessment", which enable practitioners to gauge the real value of ML methods for NID. Then, we show that the state-of-research hardly allows one to estimate the value of ML for NID. As a constructive step forward, we carry out a pragmatic assessment. We re-assess existing ML methods for NID, focusing on the classification of malicious network traffic, and consider: hundreds of configuration settings; diverse adversarial scenarios; and four hardware platforms. Our large and reproducible evaluations enable estimating the quality of ML for NID. We also validate our claims through a user-study with security practitioners.