MotionDreamer: Exploring Semantic Video Diffusion features for Zero-Shot 3D Mesh Animation

TL;DR

MotionDreamer introduces a zero-shot, mesh-based 3D animation technique that leverages video diffusion features for accurate, efficient re-animation of diverse shapes, outperforming existing methods.

Contribution

The paper presents a novel zero-shot re-animation approach using diffusion features, focusing solely on motion and compatible with standard graphics pipelines.

Findings

Outperforms existing techniques in user studies

Effective across multiple diffusion and animation models

Enhances motion fitting accuracy with diffusion features

Abstract

Animation techniques bring digital 3D worlds and characters to life. However, manual animation is tedious and automated techniques are often specialized to narrow shape classes. In our work, we propose a technique for automatic re-animation of various 3D shapes based on a motion prior extracted from a video diffusion model. Unlike existing 4D generation methods, we focus solely on the motion, and we leverage an explicit mesh-based representation compatible with existing computer-graphics pipelines. Furthermore, our utilization of diffusion features enhances accuracy of our motion fitting. We analyze efficacy of these features for animation fitting and we experimentally validate our approach for two different diffusion models and four animation models. Finally, we demonstrate that our time-efficient zero-shot method achieves a superior performance re-animating a diverse set of 3D shapes when compared to existing techniques in a user study. The project website is located at https://lukas.uzolas.com/MotionDreamer.