CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL

TL;DR

CreFlow introduces a novel reinforcement learning framework that improves embodied video generation by incorporating compositional constraints and localized corrections, significantly enhancing task success rates.

Contribution

The paper presents CreFlow, a new RL method with a credit-aware loss and corrective reflow loss, enabling better alignment with task specifications and improved performance in manipulation tasks.

Findings

CreFlow outperforms existing methods in reward alignment with human and simulator labels.

Achieves a 23.8 percentage point increase in downstream task success across eight manipulation tasks.

Uses compositional Linear Temporal Logic constraints for more faithful reward modeling.

Abstract

Video generation models trained on heterogeneous data with likelihood-surrogate objectives can produce visually plausible rollouts that violate physical constraints in embodied manipulation. Although reinforcement-learning post-training offers a natural route to adapting VGMs, existing video-RL rewards often reduce each rollout to a low-level visual metric, whereas manipulation video evaluation requires logic-based verification of whether the rollout satisfies a compositional task specification. To fill this gap, we introduce a compositional constraint-based reward model for post-training embodied video generation models, which automatically formulates task requirements as a composition of Linear Temporal Logic constraints, providing faithful rewards and localized error information in generated videos. To achieve effective improvement in high-dimensional video generation using these reward signals, we further propose CreFlow, a novel online RL framework with two key designs: i) a credit-aware NFT loss that confines the RL update to reward-relevant regions, preventing perturbations to unrelated regions during post-training; and ii) a corrective reflow loss that leverages within-group positive samples as an explicit estimate of the correction direction, stabilizing and accelerating training. Experiments show that CreFlow yields reward judgments better aligned with human and simulator success labels than existing methods and improves downstream execution success by 23.8 percentage points across eight bimanual manipulation tasks.