Flow-GRPO: Training Flow Matching Models via Online RL

TL;DR

Flow-GRPO introduces a novel method combining online reinforcement learning with flow matching models, improving sampling efficiency and generation accuracy in text-to-image tasks through innovative ODE-to-SDE conversion and denoising strategies.

Contribution

It is the first to integrate online policy gradient RL into flow matching models, enhancing sampling efficiency and generation quality in text-to-image applications.

Findings

Significant accuracy improvements in compositional generation and visual text rendering.

Enhanced human preference alignment with minimal reward hacking.

Effective application across multiple text-to-image tasks.

Abstract

We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into an equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks. For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from $63\%$ to $95\%$. In visual text rendering, accuracy improves from $59\%$ to $92\%$, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.