Towards motion from video diffusion models

TL;DR

This paper explores the potential of text-conditioned video diffusion models to generate realistic human body animations by deforming SMPL-X representations guided by score distillation sampling, revealing their capabilities and limitations.

Contribution

It introduces a method to synthesize human motion using video diffusion models and analyzes their effectiveness in capturing realistic human movements.

Findings

Models can generate plausible human motions with certain limitations.

Score distillation sampling effectively guides motion synthesis.

Insights into the potential and constraints of current diffusion models for animation.

Abstract

Text-conditioned video diffusion models have emerged as a powerful tool in the realm of video generation and editing. But their ability to capture the nuances of human movement remains under-explored. Indeed the ability of these models to faithfully model an array of text prompts can lead to a wide host of applications in human and character animation. In this work, we take initial steps to investigate whether these models can effectively guide the synthesis of realistic human body animations. Specifically we propose to synthesize human motion by deforming an SMPL-X body representation guided by Score distillation sampling (SDS) calculated using a video diffusion model. By analyzing the fidelity of the resulting animations, we gain insights into the extent to which we can obtain motion using publicly available text-to-video diffusion models using SDS. Our findings shed light on the potential and limitations of these models for generating diverse and plausible human motions, paving the way for further research in this exciting area.