VividDream: Generating 3D Scene with Ambient Dynamics
Yao-Chih Lee, Yi-Ting Chen, Andrew Wang, Ting-Hsuan Liao, Brandon Y., Feng, Jia-Bin Huang

TL;DR
VividDream is a novel method that creates explorable 4D scenes with ambient dynamics from a single image or text prompt, enabling immersive free-view exploration of dynamic 3D environments.
Contribution
It introduces a new pipeline combining static 3D scene expansion, animated video ensemble generation, and 4D scene optimization from minimal input data.
Findings
Generates compelling 4D scenes from diverse images and prompts.
Enables free-view exploration with plausible ambient dynamics.
Outperforms existing methods in scene plausibility and dynamic consistency.
Abstract
We introduce VividDream, a method for generating explorable 4D scenes with ambient dynamics from a single input image or text prompt. VividDream first expands an input image into a static 3D point cloud through iterative inpainting and geometry merging. An ensemble of animated videos is then generated using video diffusion models with quality refinement techniques and conditioned on renderings of the static 3D scene from the sampled camera trajectories. We then optimize a canonical 4D scene representation using an animated video ensemble, with per-video motion embeddings and visibility masks to mitigate inconsistencies. The resulting 4D scene enables free-view exploration of a 3D scene with plausible ambient scene dynamics. Experiments demonstrate that VividDream can provide human viewers with compelling 4D experiences generated based on diverse real images and text prompts.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputer Graphics and Visualization Techniques · Human Motion and Animation · Video Surveillance and Tracking Methods
MethodsInpainting · Diffusion
