Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes

TL;DR

This paper introduces C-PG, a policy gradient algorithm with convergence guarantees for constrained Markov Decision Processes, capable of learning deterministic policies from stochastic ones and validated on control tasks.

Contribution

The paper presents C-PG, an exploration-agnostic policy gradient method with global convergence guarantees, and demonstrates its effectiveness in learning deterministic policies in constrained settings.

Findings

C-PG achieves global last-iterate convergence under gradient domination.

C-PG effectively learns deterministic policies from stochastic hyperpolicies.

Empirical results show C-PG outperforms state-of-the-art baselines on control tasks.

Abstract

Constrained Reinforcement Learning (CRL) addresses sequential decision-making problems where agents are required to achieve goals by maximizing the expected return while meeting domain-specific constraints. In this setting, policy-based methods are widely used thanks to their advantages when dealing with continuous-control problems. These methods search in the policy space with an action-based or a parameter-based exploration strategy, depending on whether they learn the parameters of a stochastic policy or those of a stochastic hyperpolicy. We introduce an exploration-agnostic algorithm, called C-PG, which enjoys global last-iterate convergence guarantees under gradient domination assumptions. Furthermore, under specific noise models where the (hyper)policy is expressed as a stochastic perturbation of the actions or of the parameters of an underlying deterministic policy, we additionally establish global last-iterate convergence guarantees of C-PG to the optimal deterministic policy. This holds when learning a stochastic (hyper)policy and subsequently switching off the stochasticity at the end of training, thereby deploying a deterministic policy. Finally, we empirically validate both the action-based (C-PGAE) and parameter-based (C-PGPE) variants of C-PG on constrained control tasks, and compare them against state-of-the-art baselines, demonstrating their effectiveness, in particular when deploying deterministic policies after training.