Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models

TL;DR

Neighbor GRPO introduces a novel deterministic ODE-based contrastive policy optimization method that improves efficiency, convergence, and quality in generative modeling without relying on stochastic differential equations.

Contribution

It proposes Neighbor GRPO, a new SDE-free alignment algorithm that enhances flow model training by leveraging contrastive learning and theoretical policy gradient connections.

Findings

Outperforms SDE-based methods in training speed and quality

Maintains efficiency and compatibility with high-order solvers

Demonstrates superior convergence and generation results

Abstract

Group Relative Policy Optimization (GRPO) has shown promise in aligning image and video generative models with human preferences. However, applying it to modern flow matching models is challenging because of its deterministic sampling paradigm. Current methods address this issue by converting Ordinary Differential Equations (ODEs) to Stochastic Differential Equations (SDEs), which introduce stochasticity. However, this SDE-based GRPO suffers from issues of inefficient credit assignment and incompatibility with high-order solvers for fewer-step sampling. In this paper, we first reinterpret existing SDE-based GRPO methods from a distance optimization perspective, revealing their underlying mechanism as a form of contrastive learning. Based on this insight, we propose Neighbor GRPO, a novel alignment algorithm that completely bypasses the need for SDEs. Neighbor GRPO generates a diverse set of candidate trajectories by perturbing the initial noise conditions of the ODE and optimizes the model using a softmax distance-based surrogate leaping policy. We establish a theoretical connection between this distance-based objective and policy gradient optimization, rigorously integrating our approach into the GRPO framework. Our method fully preserves the advantages of deterministic ODE sampling, including efficiency and compatibility with high-order solvers. We further introduce symmetric anchor sampling for computational efficiency and group-wise quasi-norm reweighting to address reward flattening. Extensive experiments demonstrate that Neighbor GRPO significantly outperforms SDE-based counterparts in terms of training cost, convergence speed, and generation quality.