OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models

TL;DR

OP-GRPO introduces an off-policy training framework for flow-matching models, significantly improving training efficiency while maintaining high-quality generation by reusing high-quality trajectories and correcting distribution shifts.

Contribution

It is the first off-policy GRPO framework for flow-matching models, incorporating trajectory replay, importance sampling correction, and trajectory truncation for improved efficiency.

Findings

OP-GRPO achieves comparable or better performance than Flow-GRPO.

Training efficiency is improved by reducing training steps by 65.8%.

The method maintains generation quality across image and video benchmarks.

Abstract

Post training via GRPO has demonstrated remarkable effectiveness in improving the generation quality of flow-matching models. However, GRPO suffers from inherently low sample efficiency due to its on-policy training paradigm. To address this limitation, we present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models. First, we actively select high-quality trajectories and adaptively incorporate them into a replay buffer for reuse in subsequent training iterations. Second, to mitigate the distribution shift introduced by off-policy samples, we propose a sequence-level importance sampling correction that preserves the integrity of GRPO's clipping mechanism while ensuring stable policy updates. Third, we theoretically and empirically show that late denoising steps yield ill-conditioned off-policy ratios, and mitigate this by truncating trajectories at late steps. Across image and video generation benchmarks, OP-GRPO achieves comparable or superior performance to Flow-GRPO with only 34.2% of the training steps on average, yielding substantial gains in training efficiency while maintaining generation quality.