Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models

TL;DR

This paper introduces a diffusion model-based approach for synthesizing human motion from audio, enabling high-quality, style-controllable gesture, dance, and locomotion generation with flexible style interpolation.

Contribution

It adapts the DiffWave architecture with Conformers for 3D pose synthesis and extends guidance techniques for style control and ensemble modeling, advancing audio-driven motion synthesis.

Findings

Achieves top-tier motion quality in gesture and dance generation

Enables adjustable stylistic expression in synthesized motion

Demonstrates effective style interpolation using ensemble models

Abstract

Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models. We show that these models are an excellent fit for synthesising human motion that co-occurs with audio, e.g., dancing and co-speech gesticulation, since motion is complex and highly ambiguous given audio, calling for a probabilistic description. Specifically, we adapt the DiffWave architecture to model 3D pose sequences, putting Conformers in place of dilated convolutions for improved modelling power. We also demonstrate control over motion style, using classifier-free guidance to adjust the strength of the stylistic expression. Experiments on gesture and dance generation confirm that the proposed method achieves top-of-the-line motion quality, with distinctive styles whose expression can be made more or less pronounced. We also synthesise path-driven locomotion using the same model architecture. Finally, we generalise the guidance procedure to obtain product-of-expert ensembles of diffusion models and demonstrate how these may be used for, e.g., style interpolation, a contribution we believe is of independent interest. See https://www.speech.kth.se/research/listen-denoise-action/ for video examples, data, and code.