Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees

TL;DR

This paper introduces a new algorithm for robust constrained Markov decision processes that guarantees iteration complexity and improves computational efficiency over existing methods, ensuring safe and optimal policies under model uncertainty.

Contribution

It proposes a novel technique for robust policy optimization in RCMDPs that avoids binary search and provides iteration complexity guarantees, enhancing efficiency and robustness.

Findings

Algorithm achieves $ ext{O}(rac{1}{ ext{epsilon}^2})$ iteration complexity.

Reduces computation time by at least 4x for smaller discount factors.

Reduces computation time by at least 6x for larger discount factors.

Abstract

Constrained decision-making is essential for designing safe policies in real-world control systems, yet simulated environments often fail to capture real-world adversities. We consider the problem of learning a policy that will maximize the cumulative reward while satisfying a constraint, even when there is a mismatch between the real model and an accessible simulator/nominal model. In particular, we consider the robust constrained Markov decision problem (RCMDP) where an agent needs to maximize the reward and satisfy the constraint against the worst possible stochastic model under the uncertainty set centered around an unknown nominal model. Primal-dual methods, effective for standard constrained MDP (CMDP), are not applicable here because of the lack of the strong duality property. Further, one cannot apply the standard robust value-iteration based approach on the composite value function either as the worst case models may be different for the reward value function and the constraint value function. We propose a novel technique that effectively minimizes the constraint value function--to satisfy the constraints; on the other hand, when all the constraints are satisfied, it can simply maximize the robust reward value function. We prove that such an algorithm finds a policy with at most $\epsilon$ sub-optimality and feasible policy after $O(\epsilon^{-2})$ iterations. In contrast to the state-of-the-art method, we do not need to employ a binary search, thus, we reduce the computation time by at least 4x for smaller value of discount factor ($\gamma$) and by at least 6x for larger value of $\gamma$.