Efficient Action Robust Reinforcement Learning with Probabilistic Policy Execution Uncertainty

TL;DR

This paper introduces a new robust reinforcement learning framework that accounts for probabilistic action execution uncertainty, providing theoretical guarantees and an efficient algorithm that outperforms existing methods in robustness and convergence.

Contribution

It establishes the existence of optimal policies under probabilistic action uncertainty and proposes ARRLC, an algorithm with minimax optimal regret and sample complexity.

Findings

ARRLC outperforms non-robust RL algorithms in robustness.

ARRLC converges faster than robust TD in experiments.

Theoretical guarantees for optimal policies under probabilistic action uncertainty.

Abstract

Robust reinforcement learning (RL) aims to find a policy that optimizes the worst-case performance in the face of uncertainties. In this paper, we focus on action robust RL with the probabilistic policy execution uncertainty, in which, instead of always carrying out the action specified by the policy, the agent will take the action specified by the policy with probability $1-\rho$ and an alternative adversarial action with probability $\rho$. We establish the existence of an optimal policy on the action robust MDPs with probabilistic policy execution uncertainty and provide the action robust Bellman optimality equation for its solution. Furthermore, we develop Action Robust Reinforcement Learning with Certificates (ARRLC) algorithm that achieves minimax optimal regret and sample complexity. Furthermore, we conduct numerical experiments to validate our approach's robustness, demonstrating that ARRLC outperforms non-robust RL algorithms and converges faster than the robust TD algorithm in the presence of action perturbations.