Covert Adversarial Actuators in Finite MDPs

TL;DR

This paper studies covert adversarial actions in finite MDPs, analyzing how an actuator can secretly deviate from prescribed actions to minimize rewards without detection, and deriving optimal strategies and detection error exponents.

Contribution

It formulates conditions and optimization problems for covert adversarial behavior in MDPs, including asymptotic detection error analysis and strategies for the adversary.

Findings

Established conditions for covert adversarial behavior.

Derived asymptotic error exponents for detection.

Proposed optimization for adversarial performance in composite hypothesis testing.

Abstract

We consider a Markov decision process (MDP) in which actions prescribed by the controller are executed by a separate actuator, which may behave adversarially. At each time step, the controller selects and transmits an action to the actuator; however, the actuator may deviate from the intended action to degrade the control reward. Given that the controller observes only the sequence of visited states, we investigate whether the actuator can covertly deviate from the controller's policy to minimize its reward without being detected. We establish conditions for covert adversarial behavior over an infinite time horizon and formulate an optimization problem to determine the optimal adversarial policy under these conditions. Additionally, we derive the asymptotic error exponents for detection in two scenarios: (1) a binary hypothesis testing framework, where the actuator either follows the prescribed policy or a known adversarial strategy, and (2) a composite hypothesis testing framework, where the actuator may employ any stationary policy. For the latter case, we also propose an optimization problem to maximize the adversary's performance.