Balancing Constraints and Rewards with Meta-Gradient D4PG

TL;DR

This paper introduces a meta-gradient based soft-constrained reinforcement learning method that balances maximizing return with minimizing constraint violations, effectively handling complex constraints in real-world applications.

Contribution

It proposes a novel meta-gradient approach for soft-constrained RL that adapts to complex constraints without requiring precise threshold settings.

Findings

Outperforms baselines in four MuJoCo domains

Effectively balances return and constraint violations

Demonstrates robustness in real-world constraint scenarios

Abstract

Deploying Reinforcement Learning (RL) agents to solve real-world applications often requires satisfying complex system constraints. Often the constraint thresholds are incorrectly set due to the complex nature of a system or the inability to verify the thresholds offline (e.g, no simulator or reasonable offline evaluation procedure exists). This results in solutions where a task cannot be solved without violating the constraints. However, in many real-world cases, constraint violations are undesirable yet they are not catastrophic, motivating the need for soft-constrained RL approaches. We present a soft-constrained RL approach that utilizes meta-gradients to find a good trade-off between expected return and minimizing constraint violations. We demonstrate the effectiveness of this approach by showing that it consistently outperforms the baselines across four different MuJoCo domains.