Provably Efficient Sample Complexity for Robust CMDP

TL;DR

This paper introduces a new sample complexity guarantee for robust constrained Markov decision processes (RCMDPs), proposing an augmented state space and a novel value iteration algorithm that ensures near-optimal policies with safety constraints.

Contribution

The paper presents the first sample complexity guarantee for RCMDPs, introducing an augmented state space and a robust constrained value iteration algorithm.

Findings

Achieves a sample complexity of |S||A|H^5/psilon^2 with psilon violation.

Demonstrates the effectiveness of the proposed algorithm through empirical validation.

Highlights that Markovian policies may be suboptimal in RCMDPs under certain uncertainty sets.

Abstract

We study the problem of learning policies that maximize cumulative reward while satisfying safety constraints, even when the real environment differs from a simulator or nominal model. We focus on robust constrained Markov decision processes (RCMDPs), where the agent must maximize reward while ensuring cumulative utility exceeds a threshold under the worst-case dynamics within an uncertainty set. While recent works have established finite-time iteration complexity guarantees for RCMDPs using policy optimization, their sample complexity guarantees remain largely unexplored. In this paper, we first show that Markovian policies may fail to be optimal even under rectangular uncertainty sets unlike the {\em unconstrained} robust MDP. To address this, we introduce an augmented state space that incorporates the remaining utility budget into the state representation. Building on this formulation, we propose a novel Robust constrained Value iteration (RCVI) algorithm with a sample complexity of $\mathcal{\tilde{O}}(|S||A|H^5/\epsilon^2)$ achieving at most $\epsilon$ violation using a generative model where $|S|$ and $|A|$ denote the sizes of the state and action spaces, respectively, and $H$ is the episode length. To the best of our knowledge, this is the {\em first sample complexity guarantee} for RCMDP. Empirical results further validate the effectiveness of our approach.