Optimal Strong Regret and Violation in Constrained MDPs via Policy Optimization

TL;DR

This paper introduces an efficient policy optimization algorithm for constrained MDPs that achieves the optimal sublinear strong regret and violation bounds of rac{1}{2} d with a primal-dual scheme, improving upon previous methods.

Contribution

It presents the first policy optimization method achieving optimal rac{1}{2} d bounds for strong regret and violation in constrained MDPs.

Findings

Achieves rac{1}{2} d d bounds for strong regret and violation.

Uses a primal-dual scheme with policy optimization and UCB-like dual updates.

Outperforms previous algorithms with suboptimal bounds.

Abstract

We study online learning in \emph{constrained MDPs} (CMDPs), focusing on the goal of attaining sublinear strong regret and strong cumulative constraint violation. Differently from their standard (weak) counterparts, these metrics do not allow negative terms to compensate positive ones, raising considerable additional challenges. Efroni et al. (2020) were the first to propose an algorithm with sublinear strong regret and strong violation, by exploiting linear programming. Thus, their algorithm is highly inefficient, leaving as an open problem achieving sublinear bounds by means of policy optimization methods, which are much more efficient in practice. Very recently, Muller et al. (2024) have partially addressed this problem by proposing a policy optimization method that allows to attain $\widetilde{\mathcal{O}}(T^{0.93})$ strong regret/violation. This still leaves open the question of whether optimal bounds are achievable by using an approach of this kind. We answer such a question affirmatively, by providing an efficient policy optimization algorithm with $\widetilde{\mathcal{O}}(\sqrt{T})$ strong regret/violation. Our algorithm implements a primal-dual scheme that employs a state-of-the-art policy optimization approach for adversarial (unconstrained) MDPs as primal algorithm, and a UCB-like update for dual variables.