StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generation

TL;DR

StarGen introduces a novel autoregressive framework utilizing a video diffusion model for scalable, long-range, and controllable scene generation with improved consistency and pose accuracy.

Contribution

It presents a new autoregressive approach with a pre-trained video diffusion model for long-range scene generation, enhancing scalability and control.

Findings

Outperforms state-of-the-art in scalability and fidelity.

Achieves high pose accuracy in scene generation.

Supports diverse tasks like view interpolation and city layout generation.

Abstract

Recent advances in large reconstruction and generative models have significantly improved scene reconstruction and novel view generation. However, due to compute limitations, each inference with these large models is confined to a small area, making long-range consistent scene generation challenging. To address this, we propose StarGen, a novel framework that employs a pre-trained video diffusion model in an autoregressive manner for long-range scene generation. The generation of each video clip is conditioned on the 3D warping of spatially adjacent images and the temporally overlapping image from previously generated clips, improving spatiotemporal consistency in long-range scene generation with precise pose control. The spatiotemporal condition is compatible with various input conditions, facilitating diverse tasks, including sparse view interpolation, perpetual view generation, and layout-conditioned city generation. Quantitative and qualitative evaluations demonstrate StarGen's superior scalability, fidelity, and pose accuracy compared to state-of-the-art methods. Project page: https://zju3dv.github.io/StarGen.