TeMuDance: Contrastive Alignment-Based Textual Control for Music-Driven Dance Generation

TL;DR

TeMuDance is a novel framework enabling natural language control over music-driven dance generation by aligning disjoint datasets in a shared semantic space, without needing manually annotated triplets.

Contribution

It introduces a motion-centered bridging paradigm and a lightweight text control branch to enhance semantic controllability in dance generation without extensive labeled data.

Findings

TeMuDance achieves high-quality dance generation with improved text control.

The framework effectively aligns music, text, and motion in a shared embedding space.

Experimental results show competitive dance quality and enhanced semantic controllability.

Abstract

Existing music-driven dance generation approaches have achieved strong realism and effective audio-motion alignment. However, they generally lack semantic controllability, making it difficult to guide specific movements through natural language descriptions. This limitation primarily stems from the absence of large-scale datasets that jointly align music, text, and motion for supervised learning of text-conditioned control. To address this challenge, we propose TeMuDance, a framework that enables text-based control for music-conditioned dance generation without requiring any manually annotated music-text-motion triplet dataset. TeMuDance introduces a motion-centred bridging paradigm that leverages motion as a shared semantic anchor to align disjoint music-dance and text-motion datasets within a unified embedding space, enabling cross-modal retrieval of missing modalities for end-to-end training. A lightweight text control branch is then trained on top of a frozen music-to-dance diffusion backbone, preserving rhythmic fidelity while enabling fine-grained semantic guidance. To further suppress noise inherent in the retrieved supervision, we design a dual-stream fine-tuning strategy with confidence-based filtering. We also propose a novel task-aligned metric that quantifies whether textual prompts induce the intended kinematic attributes under music conditioning. Extensive experiments demonstrate that TeMuDance achieves competitive dance quality while substantially improving text-conditioned control over existing methods.