Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video

TL;DR

DriveAnyMesh introduces a 4D diffusion model that efficiently generates mesh animations from monocular videos, improving compatibility with modern rendering engines and enabling rapid, high-quality animations for complex motions.

Contribution

The paper presents a novel 4D diffusion approach using a transformer-based VAE to produce mesh animations from video, addressing efficiency and generalization issues of prior methods.

Findings

Rapid generation of high-quality mesh animations

Compatibility with modern rendering engines

Effective handling of complex motions

Abstract

We propose DriveAnyMesh, a method for driving mesh guided by monocular video. Current 4D generation techniques encounter challenges with modern rendering engines. Implicit methods have low rendering efficiency and are unfriendly to rasterization-based engines, while skeletal methods demand significant manual effort and lack cross-category generalization. Animating existing 3D assets, instead of creating 4D assets from scratch, demands a deep understanding of the input's 3D structure. To tackle these challenges, we present a 4D diffusion model that denoises sequences of latent sets, which are then decoded to produce mesh animations from point cloud trajectory sequences. These latent sets leverage a transformer-based variational autoencoder, simultaneously capturing 3D shape and motion information. By employing a spatiotemporal, transformer-based diffusion model, information is exchanged across multiple latent frames, enhancing the efficiency and generalization of the generated results. Our experimental results demonstrate that DriveAnyMesh can rapidly produce high-quality animations for complex motions and is compatible with modern rendering engines. This method holds potential for applications in both the gaming and filming industries.