DIMO: Diverse 3D Motion Generation for Arbitrary Objects

TL;DR

DIMO introduces a generative method that creates diverse 3D object motions from a single image by embedding motion patterns into a shared latent space, enabling instant sampling and applications like interpolation and language-guided generation.

Contribution

The paper presents a novel approach that leverages video priors to generate and embed diverse 3D motions into a shared latent space from a single image, supporting instant sampling.

Findings

Enables instant sampling of diverse 3D motions.

Supports 3D motion interpolation and language-guided generation.

Achieves high diversity and realism in generated motions.

Abstract

We present DIMO, a generative approach capable of generating diverse 3D motions for arbitrary objects from a single image. The core idea of our work is to leverage the rich priors in well-trained video models to extract the common motion patterns and then embed them into a shared low-dimensional latent space. Specifically, we first generate multiple videos of the same object with diverse motions. We then embed each motion into a latent vector and train a shared motion decoder to learn the distribution of motions represented by a structured and compact motion representation, i.e., neural key point trajectories. The canonical 3D Gaussians are then driven by these key points and fused to model the geometry and appearance. During inference time with learned latent space, we can instantly sample diverse 3D motions in a single-forward pass and support several interesting applications including 3D motion interpolation and language-guided motion generation. Our project page is available at https://linzhanm.github.io/dimo.