Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment

TL;DR

This paper introduces VGPO, a novel framework that improves flow matching model alignment in image generation by anchoring values across time and group dimensions, addressing reward sparsity and stagnation issues.

Contribution

VGPO redefines value estimation in flow matching models by incorporating dense, process-aware values and absolute rewards, enhancing alignment and stability in image generation.

Findings

VGPO achieves state-of-the-art image quality on benchmarks.

VGPO improves task-specific accuracy and reduces reward hacking.

VGPO maintains stable optimization signals during training.

Abstract

Group Relative Policy Optimization (GRPO) has proven highly effective in enhancing the alignment capabilities of Large Language Models (LLMs). However, current adaptations of GRPO for the flow matching-based image generation neglect a foundational conflict between its core principles and the distinct dynamics of the visual synthesis process. This mismatch leads to two key limitations: (i) Uniformly applying a sparse terminal reward across all timesteps impairs temporal credit assignment, ignoring the differing criticality of generation phases from early structure formation to late-stage tuning. (ii) Exclusive reliance on relative, intra-group rewards causes the optimization signal to fade as training converges, leading to the optimization stagnation when reward diversity is entirely depleted. To address these limitations, we propose Value-Anchored Group Policy Optimization (VGPO), a framework that redefines value estimation across both temporal and group dimensions. Specifically, VGPO transforms the sparse terminal reward into dense, process-aware value estimates, enabling precise credit assignment by modeling the expected cumulative reward at each generative stage. Furthermore, VGPO replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal even as reward diversity declines. Extensive experiments on three benchmarks demonstrate that VGPO achieves state-of-the-art image quality while simultaneously improving task-specific accuracy, effectively mitigating reward hacking. Project webpage: https://yawen-shao.github.io/VGPO/.