Data-regularized Reinforcement Learning for Diffusion Models at Scale

TL;DR

This paper introduces DDRL, a new reinforcement learning framework for diffusion models that uses data regularization to improve alignment with human preferences, reduce reward hacking, and enhance scalability.

Contribution

The paper proposes DDRL, a novel data-regularized RL approach for diffusion models, providing theoretical robustness and empirical improvements in high-resolution video generation.

Findings

Significantly improves reward alignment in diffusion models.

Reduces reward hacking and quality degradation.

Achieves highest human preference scores in experiments.

Abstract

Aligning generative diffusion models with human preferences via reinforcement learning (RL) is critical yet challenging. Most existing algorithms are often vulnerable to reward hacking, such as quality degradation, over-stylization, or reduced diversity. Our analysis demonstrates that this can be attributed to the inherent limitations of their regularization, which provides unreliable penalties. We introduce Data-regularized Diffusion Reinforcement Learning (DDRL), a novel framework that uses the forward KL divergence to anchor the policy to an off-policy data distribution. Theoretically, DDRL enables robust, unbiased integration of RL with standard diffusion training. Empirically, this translates into a simple yet effective algorithm that combines reward maximization with diffusion loss minimization. With over a million GPU hours of experiments and ten thousand double-blind human evaluations, we demonstrate on high-resolution video generation tasks that DDRL significantly improves rewards while alleviating the reward hacking seen in baselines, achieving the highest human preference and establishing a robust and scalable paradigm for diffusion post-training.