ORV: 4D Occupancy-centric Robot Video Generation

TL;DR

ORV introduces a 4D occupancy-centric framework for robot video generation that enhances fidelity, temporal consistency, and controllability by integrating action priors and occupancy-based visual guidance, supported by a new large-scale dataset.

Contribution

The paper presents ORV, a novel 4D occupancy-based approach for robot video synthesis that improves realism and control, and curates a large-scale 4D occupancy dataset for embodied scenarios.

Findings

Achieves 18.8% lower FVD than state-of-the-art methods.

Improves success rate by 3.5% on visual planning tasks.

Enhances success rate by 6.4% on policy learning.

Abstract

Recent embodied intelligence suffers from data scarcity, while conventional simulators lack visual realism. Controllable video generation is emerging as a promising data engine, yet current action-conditioned methods still fall short: generated videos are limited in fidelity and temporal consistency, poorly aligned with controls, and often constrained to singleview settings. We attribute these issues to the representational gap between sparse control inputs and dense pixel outputs. Thus, we introduce ORV, a 4D occupancy-centric framework for robot video generation that couples action priors with occupancy-derived visual priors. Concretely, we align chunked 7-DoF actions with video latents via an Action-Expert AdaLN modulation, and inject 2D renderings of 4D semantic occupancy into the generation process as soft guidance. Meanwhile, a central obstacle is the lack of occupancy data for embodied scenarios; we therefore curate ORV-Data, a large-scale, high-quality 4D semantic occupancy dataset of robot manipulation. Across BridgeV2, DROID, and RT-1, ORV improves video generation quality and controllability, achieving 18.8% lower FVD than state of the art, +3.5% success rate on visual planning, and +6.4% success rate on policy learning. Beyond singleview generation, ORV natively supports multiview consistent synthesis and enables simulation-to-real transfer despite significant domain gaps. Code, models, and data are at: https://orangesodahub.github.io/ORV