Voyaging into Perpetual Dynamic Scenes from a Single View

TL;DR

DynamicVoyager introduces a novel method for generating perpetual, 3D consistent dynamic scenes from a single view by combining 2D outpainting with 3D ray information, enabling realistic scene outpainting and motion continuity.

Contribution

It reformulates dynamic scene generation as a scene outpainting problem enriched with 3D ray data, achieving consistent 3D motions from a single view, which was challenging for previous 2D outpainting models.

Findings

Generates perpetual scenes with consistent motions along fly-through cameras.

Enables scene control via text prompts.

Outperforms prior methods in 3D motion consistency.

Abstract

The problem of generating a perpetual dynamic scene from a single view is an important problem with widespread applications in augmented and virtual reality, and robotics. However, since dynamic scenes regularly change over time, a key challenge is to ensure that different generated views be consistent with the underlying 3D motions. Prior work learns such consistency by training on multiple views, but the generated scene regions often interpolate between training views and fail to generate perpetual views. To address this issue, we propose DynamicVoyager, which reformulates dynamic scene generation as a scene outpainting problem with new dynamic content. As 2D outpainting models struggle at generating 3D consistent motions from a single 2D view, we enrich 2D pixels with information from their 3D rays that facilitates learning of 3D motion consistency. More specifically, we first map the single-view video input to a dynamic point cloud using the estimated video depths. We then render a partial video of the point cloud from a novel view and outpaint the missing regions using ray information (e.g., the distance from a ray to the point cloud) to generate 3D consistent motions. Next, we use the outpainted video to update the point cloud, which is used for outpainting the scene from future novel views. Moreover, we can control the generated content with the input text prompt. Experiments show that our model can generate perpetual scenes with consistent motions along fly-through cameras. Project page: https://tianfr.github.io/DynamicVoyager.