dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models

TL;DR

dFlowGRPO is a reinforcement learning framework for discrete flow models that improves image generation and understanding by leveraging trajectory probabilities and transition rates.

Contribution

It introduces a unified RL approach supporting various probability paths and source distributions for discrete flow models, with applications to multimodal tasks.

Findings

Outperforms existing GRPO methods on text-to-image generation.

Achieves performance competitive with continuous flow models.

Demonstrates strong understanding capabilities.

Abstract

Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of mixture path and a masked source distribution. While several recent works have explored reinforcement learning into dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate denoising as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using FlowGRPO, while also demonstrating strong capabilities on understanding tasks.