AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization

TL;DR

AR-CoPO is a novel framework that enhances autoregressive video generation by aligning it with contrastive policy optimization, improving quality and alignment with human preferences across domains.

Contribution

It introduces chunk-level alignment and a semi-on-policy training strategy to improve autoregressive video generation and alignment with human feedback.

Findings

Improves out-of-domain generalization.

Enhances in-domain human preference alignment.

Demonstrates genuine alignment over reward hacking.

Abstract

Streaming autoregressive (AR) video generators combined with few-step distillation achieve low-latency, high-quality synthesis, yet remain difficult to align via reinforcement learning from human feedback (RLHF). Existing SDE-based GRPO methods face challenges in this setting: few-step ODEs and consistency model samplers deviate from standard flow-matching ODEs, and their short, low-stochasticity trajectories are highly sensitive to initialization noise, rendering intermediate SDE exploration ineffective. We propose AR-CoPO (AutoRegressive Contrastive Policy Optimization), a framework that adapts the Neighbor GRPO contrastive perspective to streaming AR generation. AR-CoPO introduces chunk-level alignment via a forking mechanism that constructs neighborhood candidates at a randomly selected chunk, assigns sequence-level rewards, and performs localized GRPO updates. We further propose a semi-on-policy training strategy that complements on-policy exploration with exploitation over a replay buffer of reference rollouts, improving generation quality across domains. Experiments on Self-Forcing demonstrate that AR-CoPO improves both out-of-domain generalization and in-domain human preference alignment over the baseline, providing evidence of genuine alignment rather than reward hacking.