Breaking the Curse of Repulsion: Optimistic Distributionally Robust Policy Optimization for Off-Policy Generative Recommendation

TL;DR

This paper introduces DRPO, a novel policy optimization method that effectively filters high-quality data from noisy logs, overcoming the limitations of traditional RL approaches in offline recommendation tasks.

Contribution

It formulates a distributionally robust optimization framework and proves that hard filtering is optimal for recovering high-quality behaviors in offline RL for recommendations.

Findings

DRPO outperforms existing methods on mixed-quality benchmarks.

The Divergence Theory of Repulsive Optimization explains model collapse issues.

Hard filtering precisely isolates high-quality data for improved policy learning.

Abstract

Policy-based Reinforcement Learning (RL) has established itself as the dominant paradigm in generative recommendation for optimizing sequential user interactions. However, when applied to offline historical logs, these methods suffer a critical failure: the dominance of low-quality data induces severe model collapse. We first establish the Divergence Theory of Repulsive Optimization, revealing that negative gradient updates inherently trigger exponential intensity explosion during off-policy training. This theory elucidates the inherent dilemma of existing methods, exposing their inability to reconcile variance reduction and noise imitation. To break this curse, we argue that the solution lies in rigorously identifying the latent high-quality distribution entangled within the noisy behavior policy. Accordingly, we reformulate the objective as an Optimistic Distributionally Robust Optimization (DRO) problem. Guided by this formulation, we propose Distributionally Robust Policy Optimization (DRPO). We prove that hard filtering is the exact solution to this DRO objective, enabling DRPO to optimally recover high-quality behaviors while strictly discarding divergence-inducing noise. Extensive experiments demonstrate that DRPO achieves state-of-the-art performance on mixed-quality recommendation benchmarks.