TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment

TL;DR

This paper introduces TAGRPO, a post-training framework that enhances image-to-video generation by aligning trajectories with high-reward outputs, leveraging contrastive learning and a memory bank for diversity.

Contribution

We propose TAGRPO, a novel method that applies direct trajectory alignment to improve I2V models, addressing limitations of previous techniques and boosting performance.

Findings

Significant improvements over DanceGRPO in I2V generation.

Effective use of contrastive learning for trajectory alignment.

Memory bank enhances diversity and reduces computation.

Abstract

Recent studies have demonstrated the efficacy of integrating Group Relative Policy Optimization (GRPO) into flow matching models, particularly for text-to-image and text-to-video generation. However, we find that directly applying these techniques to image-to-video (I2V) models often fails to yield consistent reward improvements. To address this limitation, we present TAGRPO, a robust post-training framework for I2V models inspired by contrastive learning. Our approach is grounded in the observation that rollout videos generated from identical initial noise provide superior guidance for optimization. Leveraging this insight, we propose a novel GRPO loss applied to intermediate latents, encouraging direct alignment with high-reward trajectories while maximizing distance from low-reward counterparts. Furthermore, we introduce a memory bank for rollout videos to enhance diversity and reduce computational overhead. Despite its simplicity, TAGRPO achieves significant improvements over DanceGRPO in I2V generation.