Fundamental Limits of Man-in-the-Middle Attack Detection in Model-Free Reinforcement Learning

TL;DR

This paper analyzes the fundamental limits of detecting man-in-the-middle attacks in model-free reinforcement learning for cyber-physical systems, proposing an order-optimal detection framework and extending it to complex attack scenarios.

Contribution

It refines the attack model, derives an optimal adversary strategy, and proves the detection scheme's order-optimality, extending the framework to asynchronous attacks.

Findings

Detection time scales linearly with adversary's learning time.

Proposed detection scheme is order-optimal in efficiency.

Framework remains effective under asynchronous and intermittent attacks.

Abstract

We consider the problem of learning-based man-in-the-middle (MITM) attacks in cyber-physical systems (CPS), and extend our previously proposed Bellman Deviation Detection (BDD) framework for model-free reinforcement learning (RL). We refine the standard MDP attack model by allowing the reward function to depend on both the current and subsequent states, thereby capturing reward variations induced by errors in the adversary's transition estimate. We also derive an optimal system-identification strategy for the adversary that minimizes detectable value deviations. Further, we prove that the agent's asymptotic learning time required to secure the system scales linearly with the adversary's learning time, and that this matches the optimal lower bound. Hence, the proposed detection scheme is order-optimal in detection efficiency. Finally, we extend the framework to asynchronous and intermittent attack scenarios, where reliable detection is preserved.