Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs

TL;DR

This paper introduces a novel deterministic policy gradient primal-dual method for solving constrained Markov decision processes in continuous spaces, with proven convergence and successful application to control problems.

Contribution

It develops the first deterministic policy search method for continuous-space constrained MDPs with convergence guarantees and practical effectiveness.

Findings

Converges at a sub-linear rate to an optimal regularized primal-dual pair.

Successfully applied to robot navigation and fluid control tasks.

Proves convergence with function approximation errors considered.

Abstract

We study the problem of computing deterministic optimal policies for constrained Markov decision processes (MDPs) with continuous state and action spaces, which are widely encountered in constrained dynamical systems. Designing deterministic policy gradient methods in continuous state and action spaces is particularly challenging due to the lack of enumerable state-action pairs and the adoption of deterministic policies, hindering the application of existing policy gradient methods. To this end, we develop a deterministic policy gradient primal-dual method to find an optimal deterministic policy with non-asymptotic convergence. Specifically, we leverage regularization of the Lagrangian of the constrained MDP to propose a deterministic policy gradient primal-dual (D-PGPD) algorithm that updates the deterministic policy via a quadratic-regularized gradient ascent step and the dual variable via a quadratic-regularized gradient descent step. We prove that the primal-dual iterates of D-PGPD converge at a sub-linear rate to an optimal regularized primal-dual pair. We instantiate D-PGPD with function approximation and prove that the primal-dual iterates of D-PGPD converge at a sub-linear rate to an optimal regularized primal-dual pair, up to a function approximation error. Furthermore, we demonstrate the effectiveness of our method in two continuous control problems: robot navigation and fluid control. This appears to be the first work that proposes a deterministic policy search method for continuous-space constrained MDPs.