Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization

TL;DR

This paper introduces Reward-Decorrelated Policy Optimization (RDPO), a novel method for stabilizing multi-task reinforcement learning by normalizing and decorrelating heterogeneous reward signals.

Contribution

RDPO explicitly addresses reward correlation and heterogeneity issues using normalization and whitening techniques, improving stability and performance in complex RL environments.

Findings

RDPO enhances instruction following, writing quality, and robustness in LongCat-Flash.

RDPO remains competitive on reasoning and coding evaluations.

RDPO stabilizes advantage allocation across diverse reward types.

Abstract

Complex reinforcement learning environments frequently employ multi-task and mixed-reward formulations. In these settings, heterogeneous reward distributions and correlated reward dimensions often destabilize the construction of scalar advantages. To address these challenges, we propose Reward-Decorrelated Policy Optimization (RDPO), a reward-processing method designed to explicitly target both failure modes. RDPO first utilizes Magnitude-Aware Quantile normalization to stabilize prompt-level advantage allocation across binary, fractional, and continuous rewards. It then applies Mahalanobis whitening within each active reward subspace to mitigate correlation redundancy prior to aggregation. When applied during the post-training of LongCat-Flash, RDPO enhances instruction following, writing quality, and robustness to hard prompts while remaining broadly competitive on reasoning and coding evaluations.