Reinforcement Learning under Threats

TL;DR

This paper introduces Threatened Markov Decision Processes (TMDPs) and a level-$k$ thinking scheme to enhance reinforcement learning robustness against adversarial threats, supported by theoretical analysis and extensive empirical validation.

Contribution

It presents a novel framework for RL under adversarial threats and a new learning scheme to effectively handle such threats, with theoretical and empirical validation.

Findings

Accounting for adversaries improves RL performance.

The proposed framework enhances robustness in security-related RL scenarios.

Empirical results demonstrate the effectiveness of the approach.

Abstract

In several reinforcement learning (RL) scenarios, mainly in security settings, there may be adversaries trying to interfere with the reward generating process. In this paper, we introduce Threatened Markov Decision Processes (TMDPs), which provide a framework to support a decision maker against a potential adversary in RL. Furthermore, we propose a level-$k$ thinking scheme resulting in a new learning framework to deal with TMDPs. After introducing our framework and deriving theoretical results, relevant empirical evidence is given via extensive experiments, showing the benefits of accounting for adversaries while the agent learns.