Sample-Efficient Constrained Reinforcement Learning with General Parameterization

TL;DR

This paper introduces a new algorithm for constrained reinforcement learning that improves sample efficiency and guarantees near-optimal solutions with respect to both reward maximization and constraint satisfaction.

Contribution

The paper proposes the PD-ANPG algorithm, achieving the first near-optimal sample complexity for general parameterized CMDPs with accelerated natural policy gradients.

Findings

Improves sample complexity by a factor of (1-γ)^{-1}

Achieves the theoretical lower bound in ε^{-1}

Ensures ε-global optimality and constraint violation bounds

Abstract

We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon while ensuring that the expected discounted sum of costs exceeds a certain threshold. Building on the idea of momentum-based acceleration, we develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an $\epsilon$ global optimality gap and $\epsilon$ constraint violation with $\tilde{\mathcal{O}}((1-\gamma)^{-7}\epsilon^{-2})$ sample complexity for general parameterized policies where $\gamma$ denotes the discount factor. This improves the state-of-the-art sample complexity in general parameterized CMDPs by a factor of $\mathcal{O}((1-\gamma)^{-1}\epsilon^{-2})$ and achieves the theoretical lower bound in $\epsilon^{-1}$.