Balancing detectability and performance of attacks on the control channel of Markov Decision Processes

TL;DR

This paper introduces a new information-theoretic approach to designing stealthy poisoning attacks on MDP control channels, balancing attack effectiveness with detectability in reinforcement learning systems.

Contribution

It proposes a novel attack formulation that considers both attack performance and detectability, addressing limitations of existing norm-based constraints.

Findings

Trade-off between attack effectiveness and detectability demonstrated

Information-theoretic measures effectively quantify attack stealthiness

Numerical simulations illustrate the balance between attack success and detection risk

Abstract

We investigate the problem of designing optimal stealthy poisoning attacks on the control channel of Markov decision processes (MDPs). This research is motivated by the recent interest of the research community for adversarial and poisoning attacks applied to MDPs, and reinforcement learning (RL) methods. The policies resulting from these methods have been shown to be vulnerable to attacks perturbing the observations of the decision-maker. In such an attack, drawing inspiration from adversarial examples used in supervised learning, the amplitude of the adversarial perturbation is limited according to some norm, with the hope that this constraint will make the attack imperceptible. However, such constraints do not grant any level of undetectability and do not take into account the dynamic nature of the underlying Markov process. In this paper, we propose a new attack formulation, based on information-theoretical quantities, that considers the objective of minimizing the detectability of the attack as well as the performance of the controlled process. We analyze the trade-off between the efficiency of the attack and its detectability. We conclude with examples and numerical simulations illustrating this trade-off.