Embedding-perturbed Exploration Preference Optimization for Flow Models

TL;DR

This paper introduces E^2PO, a novel reinforcement learning framework that maintains sample diversity through embedding perturbations, leading to more stable training and better alignment with human preferences.

Contribution

E^2PO is the first method to use embedding-level perturbations to sustain variance and improve stability in group-based optimization for generative models.

Findings

E^2PO outperforms existing methods in aligning with human preferences.

Embedding perturbations prevent variance decay and stabilize training.

Significant improvements demonstrated in extensive experiments.

Abstract

Recent advancements have established Reinforcement Learning (RL) as a pivotal paradigm for aligning generative models with human intent. However, group-based optimization frameworks (e.g., GRPO) face a critical limitation: the rapid decay of intra-group variance. As the distinctiveness among samples within a group diminishes, the variance approaches zero. This eliminates the very learning signal required for optimization, rendering the process unstable and forcing the policy into premature stagnation or reward hacking. Existing strategies, such as varying the initial noise or increasing group sizes, often fail to address this fundamental issue, resulting in training instability or diminishing returns. To overcome these challenges, we propose $\textbf{Embedding-perturbed Exploration Preference Optimization (}E^2\textbf{PO)}$, a novel framework that sustains optimization through embedding-level perturbation. Our method introduces structured, embedding-level perturbations within sample groups, guaranteeing a robust variance that preserves the discriminative signal throughout the training process. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art baselines, achieving a more faithful alignment with human preference.