Plenoptic Video Generation

TL;DR

PlenopticDreamer is a novel framework for multi-view video generation that maintains spatio-temporal coherence and achieves state-of-the-art results through a camera-guided, autoregressive, and self-conditioned approach.

Contribution

It introduces a multi-in-single-out video-conditioned model with adaptive retrieval and advanced training strategies for coherent, high-fidelity plenoptic video re-rendering.

Findings

Achieves state-of-the-art performance on Basic and Agibot benchmarks.

Demonstrates superior view synchronization and visual fidelity.

Supports diverse view transformations in complex scenarios.

Abstract

Camera-controlled generative video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain spatio-temporal memory. The core idea is to train a multi-in-single-out video-conditioned model in an autoregressive manner, aided by a camera-guided video retrieval strategy that adaptively selects salient videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-video conditioning mechanism to support extended video generation. Extensive experiments on the Basic and Agibot benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: https://research.nvidia.com/labs/dir/plenopticdreamer/