ViMo: Generating Motions from Casual Videos

TL;DR

ViMo leverages diffusion models to generate diverse, realistic 3D human motions from casual videos, overcoming challenges like camera movements and occlusions, and enabling applications like dance motion synthesis.

Contribution

Introduces ViMo, a novel video-to-motion framework that captures motions from casual videos, expanding motion generation beyond limited Mocap datasets using diffusion models.

Findings

Generates natural, diverse motions from complex videos

Handles rapid movements, varying perspectives, and occlusions effectively

Enables applications like dance motion synthesis from music and style

Abstract

Although humans have the innate ability to imagine multiple possible actions from videos, it remains an extraordinary challenge for computers due to the intricate camera movements and montages. Most existing motion generation methods predominantly rely on manually collected motion datasets, usually tediously sourced from motion capture (Mocap) systems or Multi-View cameras, unavoidably resulting in a limited size that severely undermines their generalizability. Inspired by recent advance of diffusion models, we probe a simple and effective way to capture motions from videos and propose a novel Video-to-Motion-Generation framework (ViMo) which could leverage the immense trove of untapped video content to produce abundant and diverse 3D human motions. Distinct from prior work, our videos could be more causal, including complicated camera movements and occlusions. Striking experimental results demonstrate the proposed model could generate natural motions even for videos where rapid movements, varying perspectives, or frequent occlusions might exist. We also show this work could enable three important downstream applications, such as generating dancing motions according to arbitrary music and source video style. Extensive experimental results prove that our model offers an effective and scalable way to generate diversity and realistic motions. Code and demos will be public soon.