Voyaging into Perpetual Dynamic Scenes from a Single View
Fengrui Tian, Tianjiao Ding, Jinqi Luo, Hancheng Min, Ren\'e Vidal

TL;DR
DynamicVoyager introduces a novel method for generating perpetual, 3D consistent dynamic scenes from a single view by combining 2D outpainting with 3D ray information, enabling realistic scene outpainting and motion continuity.
Contribution
It reformulates dynamic scene generation as a scene outpainting problem enriched with 3D ray data, achieving consistent 3D motions from a single view, which was challenging for previous 2D outpainting models.
Findings
Generates perpetual scenes with consistent motions along fly-through cameras.
Enables scene control via text prompts.
Outperforms prior methods in 3D motion consistency.
Abstract
The problem of generating a perpetual dynamic scene from a single view is an important problem with widespread applications in augmented and virtual reality, and robotics. However, since dynamic scenes regularly change over time, a key challenge is to ensure that different generated views be consistent with the underlying 3D motions. Prior work learns such consistency by training on multiple views, but the generated scene regions often interpolate between training views and fail to generate perpetual views. To address this issue, we propose DynamicVoyager, which reformulates dynamic scene generation as a scene outpainting problem with new dynamic content. As 2D outpainting models struggle at generating 3D consistent motions from a single 2D view, we enrich 2D pixels with information from their 3D rays that facilitates learning of 3D motion consistency. More specifically, we first map…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
Topics3D Shape Modeling and Analysis · Generative Adversarial Networks and Image Synthesis · Advanced Vision and Imaging
