The Role of Learning in Attacking ML-based Network Intrusion Detection

TL;DR

This paper introduces lightweight reinforcement learning agents that efficiently evaluate the robustness of ML-based network intrusion detection systems, including non-differentiable models, with high success rates and scalability.

Contribution

It presents a novel RL-based approach for offline training of evasion strategies that can be deployed without gradient computations, enabling scalable robustness testing.

Findings

Agents achieve up to 58.1% attack success rate.

Up to 1,042X improvement in attack throughput over gradient-based methods.

RL agents maintain effectiveness on non-differentiable models with 29.8% success.

Abstract

Machine learning (ML)-based network intrusion detection is susceptible to attacks that perturb malicious network flows to evade detection. Existing approaches to evaluating the robustness of these models rely on gradient-based optimization that are computationally expensive and restricted to differentiable model architectures. This limits their practicality for continuous, large-scale evaluation. To address this, we develop lightweight adversarial agents trained via reinforcement learning (RL) that decouples the cost of learning an evasion strategy from the cost of executing it. These agents learn offline to perturb malicious NetFlow records to evade surrogate intrusion detection models, encoding the resulting strategy into a reusable policy that requires no gradient computation at deployment. We evaluate our approach on four NetFlow datasets spanning enterprise, cloud, and IoT environments against diverse model architectures, including non-differentiable classifiers that gradient-based methods cannot evaluate directly. Agents achieve up to 58.1% attack success at 0.31ms per attack demonstrating up to 1,042X improvement in throughput (attack success per ms) over gradient-based methods. On non-differentiable targets, gradient-based methods lose over 59% of their effectiveness to surrogate transfer, while the RL agent evaluates these models directly at 29.8% attack success. We further conduct a comprehensive transferability study on ML-based intrusion detection, evaluating agent generalization across unseen model architectures and traffic distributions. Our results establish lightweight RL agents as a practical and scalable tool for continuous ML robustness evaluation across diverse network intrusion detection environments.