Euphonium: Steering Video Flow Matching via Process Reward Gradient Guided Stochastic Dynamics

TL;DR

Euphonium introduces a reward-gradient guided stochastic dynamics framework for more efficient and high-quality flow matching in video generation, significantly improving training speed and alignment with human preferences.

Contribution

The paper presents a theoretically grounded method that guides sampling with process reward gradients, internalizes guidance into the flow model, and demonstrates superior performance in text-to-video tasks.

Findings

Achieves better alignment in text-to-video generation.

Speeds up training convergence by 1.66 times.

Unifies existing sampling methods under a new theoretical framework.

Abstract

While online Reinforcement Learning has emerged as a crucial technique for aligning flow matching models with human preferences, current approaches are hindered by inefficient exploration during training rollouts. Relying on undirected stochasticity and sparse outcome rewards, these methods struggle to discover high-reward samples, resulting in data-inefficient and slow optimization. To address these limitations, we propose Euphonium, a novel framework that steers generation via process reward gradient guided dynamics. Our key insight is to formulate the sampling process as a theoretically principled Stochastic Differential Equation that explicitly incorporates the gradient of a Process Reward Model into the flow drift. This design enables dense, step-by-step steering toward high-reward regions, advancing beyond the unguided exploration in prior works, and theoretically encompasses existing sampling methods (e.g., Flow-GRPO, DanceGRPO) as special cases. We further derive a distillation objective that internalizes the guidance signal into the flow network, eliminating inference-time dependency on the reward model. We instantiate this framework with a Dual-Reward Group Relative Policy Optimization algorithm, combining latent process rewards for efficient credit assignment with pixel-level outcome rewards for final visual fidelity. Experiments on text-to-video generation show that Euphonium achieves better alignment compared to existing methods while accelerating training convergence by 1.66x. Our code is available at https://github.com/zerzerzerz/Euphonium