Policy Optimization for Constrained MDPs with Provable Fast Global Convergence

TL;DR

This paper introduces the PMD-PD algorithm, a policy gradient method for constrained MDPs that provably converges faster than previous approaches, with experimental validation showing improved performance.

Contribution

The paper proposes a novel policy mirror descent-primal dual algorithm that achieves a faster $ ext{O}(rac{ ext{log}(T)}{T})$ convergence rate for constrained MDPs, improving upon prior $ ext{O}(rac{1}{ ext{sqrt}(T)})$ results.

Findings

Faster $ ext{O}(rac{ ext{log}(T)}{T})$ convergence rate achieved.

Algorithm outperforms existing policy gradient methods in experiments.

Extensions handle zero constraint violation and sample-based estimation.

Abstract

We address the problem of finding the optimal policy of a constrained Markov decision process (CMDP) using a gradient descent-based algorithm. Previous results have shown that a primal-dual approach can achieve an $\mathcal{O}(1/\sqrt{T})$ global convergence rate for both the optimality gap and the constraint violation. We propose a new algorithm called policy mirror descent-primal dual (PMD-PD) algorithm that can provably achieve a faster $\mathcal{O}(\log(T)/T)$ convergence rate for both the optimality gap and the constraint violation. For the primal (policy) update, the PMD-PD algorithm utilizes a modified value function and performs natural policy gradient steps, which is equivalent to a mirror descent step with appropriate regularization. For the dual update, the PMD-PD algorithm uses modified Lagrange multipliers to ensure a faster convergence rate. We also present two extensions of this approach to the settings with zero constraint violation and sample-based estimation. Experimental results demonstrate the faster convergence rate and the better performance of the PMD-PD algorithm compared with existing policy gradient-based algorithms.