Birth and Death of a Rose

TL;DR

This paper presents a novel method for generating temporally consistent 3D object sequences, like blooming roses, from pre-trained 2D diffusion models, enabling controllable rendering from any viewpoint and lighting.

Contribution

We introduce Neural Templates for temporal-state-guided distillation, allowing automatic extraction of dynamic object intrinsics from 2D models, reducing manual effort in 3D animation.

Findings

High-quality temporal object intrinsics generated for natural phenomena

Enables controllable rendering from any viewpoint and lighting

Supports diverse dynamic objects like blooming roses

Abstract

We study the problem of generating temporal object intrinsics -- temporally evolving sequences of object geometry, reflectance, and texture, such as a blooming rose -- from pre-trained 2D foundation models. Unlike conventional 3D modeling and animation techniques that require extensive manual effort and expertise, we introduce a method that generates such assets with signals distilled from pre-trained 2D diffusion models. To ensure the temporal consistency of object intrinsics, we propose Neural Templates for temporal-state-guided distillation, derived automatically from image features from self-supervised learning. Our method can generate high-quality temporal object intrinsics for several natural phenomena and enable the sampling and controllable rendering of these dynamic objects from any viewpoint, under any environmental lighting conditions, at any time of their lifespan. Project website: https://chen-geng.com/rose4d