Constrained Reinforcement Learning via Dissipative Saddle Flow Dynamics

TL;DR

This paper introduces a new constrained reinforcement learning algorithm based on dissipative saddle flow dynamics, ensuring almost sure convergence to the optimal policy without the limitations of previous stochastic gradient methods.

Contribution

It proposes a novel stochastic gradient descent-ascent algorithm for constrained RL that guarantees convergence to the optimal policy using saddle-flow dynamics.

Findings

Algorithm converges almost surely to the optimal policy.

Addresses limitations of previous primal-dual methods.

Provides a theoretically grounded approach for constrained RL.

Abstract

In constrained reinforcement learning (C-RL), an agent seeks to learn from the environment a policy that maximizes the expected cumulative reward while satisfying minimum requirements in secondary cumulative reward constraints. Several algorithms rooted in sampled-based primal-dual methods have been recently proposed to solve this problem in policy space. However, such methods are based on stochastic gradient descent ascent algorithms whose trajectories are connected to the optimal policy only after a mixing output stage that depends on the algorithm's history. As a result, there is a mismatch between the behavioral policy and the optimal one. In this work, we propose a novel algorithm for constrained RL that does not suffer from these limitations. Leveraging recent results on regularized saddle-flow dynamics, we develop a novel stochastic gradient descent-ascent algorithm whose trajectories converge to the optimal policy almost surely.