Game-Theoretic Neyman-Pearson Detection to Combat Strategic Evasion

TL;DR

This paper develops a game-theoretic framework for Neyman-Pearson detection to effectively counter strategic evasive attacks in network security, demonstrating improved detection performance through equilibrium analysis.

Contribution

It introduces a novel game-theoretic approach to Neyman-Pearson detection, modeling attacker-detector interactions and deriving equilibrium-based detection strategies.

Findings

Evasion-aware detectors outperform passive detectors in strategic scenarios.

Equilibrium ROC curves characterize the performance of attacker-detector pairs.

The framework extends to sequential message settings with analytical validation.

Abstract

The security in networked systems depends greatly on recognizing and identifying adversarial behaviors. Traditional detection methods focus on specific categories of attacks and have become inadequate for increasingly stealthy and deceptive attacks that are designed to bypass detection strategically. This work aims to develop a holistic theory to countermeasure such evasive attacks. We focus on extending a fundamental class of statistical-based detection methods based on Neyman-Pearson's (NP) hypothesis testing formulation. We propose game-theoretic frameworks to capture the conflicting relationship between a strategic evasive attacker and an evasion-aware NP detector. By analyzing both the equilibrium behaviors of the attacker and the NP detector, we characterize their performance using Equilibrium Receiver-Operational-Characteristic (EROC) curves. We show that the evasion-aware NP detectors outperform the passive ones in the way that the former can act strategically against the attacker's behavior and adaptively modify their decision rules based on the received messages. In addition, we extend our framework to a sequential setting where the user sends out identically distributed messages. We corroborate the analytical results with a case study of anomaly detection.