Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes

TL;DR

Gaussians2Life introduces a novel method that combines video diffusion models with 3D Gaussian Splatting to animate static 3D scenes, creating realistic and consistent 3D animations from 2D videos.

Contribution

The paper presents a new approach that leverages video diffusion models to animate complex 3D scenes in Gaussian Splatting representation, enabling multi-object and multi-scene animations.

Findings

Enables realistic animations of complex 3D scenes.

Supports animation of a wide variety of object classes.

Creates consistent and immersive 3D experiences.

Abstract

State-of-the-art novel view synthesis methods achieve impressive results for multi-view captures of static 3D scenes. However, the reconstructed scenes still lack "liveliness," a key component for creating engaging 3D experiences. Recently, novel video diffusion models generate realistic videos with complex motion and enable animations of 2D images, however they cannot naively be used to animate 3D scenes as they lack multi-view consistency. To breathe life into the static world, we propose Gaussians2Life, a method for animating parts of high-quality 3D scenes in a Gaussian Splatting representation. Our key idea is to leverage powerful video diffusion models as the generative component of our model and to combine these with a robust technique to lift 2D videos into meaningful 3D motion. We find that, in contrast to prior work, this enables realistic animations of complex, pre-existing 3D scenes and further enables the animation of a large variety of object classes, while related work is mostly focused on prior-based character animation, or single 3D objects. Our model enables the creation of consistent, immersive 3D experiences for arbitrary scenes.