TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward

TL;DR

TDM-R1 introduces a novel reinforcement learning framework that effectively incorporates non-differentiable rewards into few-step diffusion models, significantly enhancing their generative quality and alignment capabilities.

Contribution

It proposes a decoupled RL paradigm with practical methods for reward signal extraction, enabling the use of non-differentiable rewards in few-step generative models.

Findings

Achieves state-of-the-art RL performance on in-domain and out-of-domain metrics.

Outperforms existing models with only 4 NFEs on Z-Image.

Improves text-to-image quality and preference alignment.

Abstract

While few-step generative models have enabled powerful image and video generation at significantly lower cost, generic reinforcement learning (RL) paradigms for few-step models remain an unsolved problem. Existing RL approaches for few-step diffusion models strongly rely on back-propagating through differentiable reward models, thereby excluding the majority of important real-world reward signals, e.g., non-differentiable rewards such as humans' binary likeness, object counts, etc. To properly incorporate non-differentiable rewards to improve few-step generative models, we introduce TDM-R1, a novel reinforcement learning paradigm built upon a leading few-step model, Trajectory Distribution Matching (TDM). TDM-R1 decouples the learning process into surrogate reward learning and generator learning. Furthermore, we developed practical methods to obtain per-step reward signals along the deterministic generation trajectory of TDM, resulting in a unified RL post-training method that significantly improves few-step models' ability with generic rewards. We conduct extensive experiments ranging from text-rendering, visual quality, and preference alignment. All results demonstrate that TDM-R1 is a powerful reinforcement learning paradigm for few-step text-to-image models, achieving state-of-the-art reinforcement learning performances on both in-domain and out-of-domain metrics. Furthermore, TDM-R1 also scales effectively to the recent strong Z-Image model, consistently outperforming both its 100-NFE and few-step variants with only 4 NFEs. Project page: https://github.com/Luo-Yihong/TDM-R1