Towards a Practical Understanding of Lagrangian Methods in Safe Reinforcement Learning

TL;DR

This paper empirically analyzes Lagrangian methods in safe reinforcement learning, revealing the sensitivity of the Lagrange multiplier and the importance of cost limit selection, supported by Pareto frontiers and an open-source benchmark.

Contribution

It provides a systematic empirical study of the trade-offs in safe RL, introduces Pareto frontiers for visualization, and offers guidelines for cost limit selection and an open-source code base.

Findings

Lagrange multiplier sensitivity varies across tasks and regimes.

Cost restrictiveness can differ within the same task.

Careful cost limit selection is crucial for evaluating safe RL methods.

Abstract

Safe reinforcement learning addresses constrained optimization problems where maximizing performance must be balanced against safety constraints, and Lagrangian methods are a widely used approach for this purpose. However, the effectiveness of Lagrangian methods depends crucially on the choice of the Lagrange multiplier $\lambda$, which governs the multi-objective trade-off between return and cost. A common practice is to update the multiplier automatically during training. Although this approach is standard in practice, there remains limited empirical evidence on the optimally achievable trade-off between return and cost as a function of $\lambda$, and there is currently no systematic benchmark comparing automated update mechanisms to this empirical optimum. Therefore, we study (i) the constraint geometry for eight widely used safety tasks and (ii) the previously overlooked constraint-regime sensitivity of different Lagrange multiplier update mechanisms in safe reinforcement learning. Through the lens of multi-objective analysis, we present empirical Pareto frontiers that offer a complete visualization of the trade-off between return and cost in the underlying optimization problem. Our results reveal the highly sensitive nature of $\lambda$ and further show that the restrictiveness of the constraint cost can vary across different cost limits within the same task. This highlights the importance of careful cost limit selection across different regions of cost restrictiveness when evaluating safe reinforcement learning methods. We provide a recommended set of cost limits for each evaluated task and offer an open-source code base: https://github.com/lindsayspoor/Lagrangian_SafeRL.