Manifold-Aware Exploration for Reinforcement Learning in Video Generation

TL;DR

This paper introduces SAGE-GRPO, a manifold-aware exploration method for reinforcement learning in video generation, which constrains exploration within the data manifold to improve stability and reward estimation accuracy.

Contribution

It proposes a novel manifold-aware exploration framework with micro and macro level constraints, including a curvature-corrected SDE and dual trust region, to enhance video generation quality.

Findings

SAGE-GRPO outperforms previous methods in reward metrics and visual quality.

The approach stabilizes training by constraining exploration near the data manifold.

Results demonstrate improved reward maximization and video quality.

Abstract

Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.