MCM: Multi-condition Motion Synthesis Framework

TL;DR

This paper introduces MCM, a multi-condition human motion synthesis framework that extends diffusion models to handle both text and audio inputs, enabling high-quality, semantically meaningful motion generation for diverse modalities.

Contribution

The paper presents a novel dual-branch diffusion framework, MCM, incorporating a Transformer-based MWNet to effectively synthesize motion from multiple conditions, including audio and text.

Findings

Achieves competitive results in single-condition HMS tasks.

Successfully extends diffusion models to multi-condition scenarios.

Maintains motion quality and semantic relevance across modalities.

Abstract

Conditional human motion synthesis (HMS) aims to generate human motion sequences that conform to specific conditions. Text and audio represent the two predominant modalities employed as HMS control conditions. While existing research has primarily focused on single conditions, the multi-condition human motion synthesis remains underexplored. In this study, we propose a multi-condition HMS framework, termed MCM, based on a dual-branch structure composed of a main branch and a control branch. This framework effectively extends the applicability of the diffusion model, which is initially predicated solely on textual conditions, to auditory conditions. This extension encompasses both music-to-dance and co-speech HMS while preserving the intrinsic quality of motion and the capabilities for semantic association inherent in the original model. Furthermore, we propose the implementation of a Transformer-based diffusion model, designated as MWNet, as the main branch. This model adeptly apprehends the spatial intricacies and inter-joint correlations inherent in motion sequences, facilitated by the integration of multi-wise self-attention modules. Extensive experiments show that our method achieves competitive results in single-condition and multi-condition HMS tasks.