Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning

TL;DR

This paper introduces CoMOGA, a gradient aggregation method for constrained multi-objective reinforcement learning that avoids gradient conflicts, ensuring stable training and constraint satisfaction across tasks.

Contribution

The paper proposes a simple yet effective gradient aggregation approach for constrained multi-objective RL that guarantees convergence and handles safety constraints.

Findings

Prevents gradient conflicts in multi-objective RL.

Ensures constraint satisfaction in experiments.

Guarantees optimal convergence in tabular settings.

Abstract

In many real-world applications, a reinforcement learning (RL) agent should consider multiple objectives and adhere to safety guidelines. To address these considerations, we propose a constrained multi-objective RL algorithm named Constrained Multi-Objective Gradient Aggregator (CoMOGA). In the field of multi-objective optimization, managing conflicts between the gradients of the multiple objectives is crucial to prevent policies from converging to local optima. It is also essential to efficiently handle safety constraints for stable training and constraint satisfaction. We address these challenges straightforwardly by treating the maximization of multiple objectives as a constrained optimization problem (COP), where the constraints are defined to improve the original objectives. Existing safety constraints are then integrated into the COP, and the policy is updated using a linear approximation, which ensures the avoidance of gradient conflicts. Despite its simplicity, CoMOGA guarantees optimal convergence in tabular settings. Through various experiments, we have confirmed that preventing gradient conflicts is critical, and the proposed method achieves constraint satisfaction across all tasks.