Enhanced Fine-grained Motion Diffusion for Text-driven Human Motion Synthesis

TL;DR

This paper introduces DiffKFC, a diffusion model for text-driven human motion synthesis that enables fine-grained, controllable animation using sparse keyframes and semantic guidance, outperforming existing methods.

Contribution

The paper presents a novel diffusion-based approach with dual-level control and customized attention modules for improved fine-grained, controllable motion synthesis from text and keyframes.

Findings

Achieves state-of-the-art semantic fidelity in motion synthesis.

Enables fine-grained control with sparse keyframes.

Produces seamless transitions in generated motions.

Abstract

The emergence of text-driven motion synthesis technique provides animators with great potential to create efficiently. However, in most cases, textual expressions only contain general and qualitative motion descriptions, while lack fine depiction and sufficient intensity, leading to the synthesized motions that either (a) semantically compliant but uncontrollable over specific pose details, or (b) even deviates from the provided descriptions, bringing animators with undesired cases. In this paper, we propose DiffKFC, a conditional diffusion model for text-driven motion synthesis with KeyFrames Collaborated, enabling realistic generation with collaborative and efficient dual-level control: coarse guidance at semantic level, with only few keyframes for direct and fine-grained depiction down to body posture level. Unlike existing inference-editing diffusion models that incorporate conditions without training, our conditional diffusion model is explicitly trained and can fully exploit correlations among texts, keyframes and the diffused target frames. To preserve the control capability of discrete and sparse keyframes, we customize dilated mask attention modules where only partial valid tokens participate in local-to-global attention, indicated by the dilated keyframe mask. Additionally, we develop a simple yet effective smoothness prior, which steers the generated frames towards seamless keyframe transitions at inference. Extensive experiments show that our model not only achieves state-of-the-art performance in terms of semantic fidelity, but more importantly, is able to satisfy animator requirements through fine-grained guidance without tedious labor.