Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization

TL;DR

This paper introduces D3PO, a PPO-based framework for multi-objective reinforcement learning that improves Pareto front discovery by decomposing optimization, stabilizing training, and encouraging diversity, outperforming prior methods.

Contribution

D3PO reorganizes multi-objective policy optimization to address gradient interference and representational collapse, enabling reliable and diverse Pareto front discovery with a single policy.

Findings

D3PO outperforms prior methods on standard MORL benchmarks.

D3PO discovers broader and higher-quality Pareto fronts.

D3PO matches or exceeds state-of-the-art hypervolume and utility metrics.

Abstract

Multi-objective reinforcement learning (MORL) seeks to learn policies that balance multiple, often conflicting objectives. Although a single preference-conditioned policy is the most flexible and scalable solution, existing approaches remain brittle in practice, frequently failing to recover complete Pareto fronts. We show that this failure stems from two structural issues in current methods: destructive gradient interference caused by premature scalarization and representational collapse across the preference space. We introduce $D^3PO$, a PPO-based framework that reorganizes multi-objective policy optimization to address these issues directly. $D^3PO$ preserves per-objective learning signals through a decomposed optimization pipeline and integrates preferences only after stabilization, enabling reliable credit assignment. In addition, a scaled diversity regularizer enforces sensitivity of policy behavior to preference changes, preventing collapse. Across standard MORL benchmarks, including high-dimensional and many-objective control tasks, $D^3PO$ consistently discovers broader and higher-quality Pareto fronts than prior single- and multi-policy methods, matching or exceeding state-of-the-art hypervolume and expected utility while using a single deployable policy.