Text-To-4D Dynamic Scene Generation

TL;DR

MAV3D introduces a novel method for generating 3D dynamic scenes from text descriptions using a 4D neural radiance field and a text-to-video diffusion model, enabling view-consistent scene visualization without 3D data.

Contribution

This work is the first to generate 3D dynamic scenes from text, leveraging a 4D NeRF optimized with a T2V diffusion model trained on 2D data.

Findings

Outperforms previous baselines in quality and consistency

Can generate scenes viewable from any angle

Does not require 3D or 4D training data

Abstract

We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions. Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency by querying a Text-to-Video (T2V) diffusion-based model. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment. MAV3D does not require any 3D or 4D data and the T2V model is trained only on Text-Image pairs and unlabeled videos. We demonstrate the effectiveness of our approach using comprehensive quantitative and qualitative experiments and show an improvement over previously established internal baselines. To the best of our knowledge, our method is the first to generate 3D dynamic scenes given a text description.