FrankenMotion: Part-level Human Motion Generation and Composition

TL;DR

FrankenMotion introduces a novel framework for fine-grained, part-level human motion generation guided by temporally-structured text prompts, enabled by a new high-quality dataset with atomic annotations, allowing enhanced controllability and motion composition.

Contribution

This work is the first to provide a dataset with atomic, temporally-aware part-level annotations and a diffusion-based model enabling detailed, controllable human motion generation from text.

Findings

Outperforms previous models in motion generation quality.

Enables composition of unseen motions during training.

Provides fine-grained control over individual body parts and actions.

Abstract

Human motion generation from text prompts has made remarkable progress in recent years. However, existing methods primarily rely on either sequence-level or action-level descriptions due to the absence of fine-grained, part-level motion annotations. This limits their controllability over individual body parts. In this work, we construct a high-quality motion dataset with atomic, temporally-aware part-level text annotations, leveraging the reasoning capabilities of large language models (LLMs). Unlike prior datasets that either provide synchronized part captions with fixed time segments or rely solely on global sequence labels, our dataset captures asynchronous and semantically distinct part movements at fine temporal resolution. Based on this dataset, we introduce a diffusion-based part-aware motion generation framework, namely FrankenMotion, where each body part is guided by its own temporally-structured textual prompt. This is, to our knowledge, the first work to provide atomic, temporally-aware part-level motion annotations and have a model that allows motion generation with both spatial (body part) and temporal (atomic action) control. Experiments demonstrate that FrankenMotion outperforms all previous baseline models adapted and retrained for our setting, and our model can compose motions unseen during training. Our code and dataset will be publicly available upon publication.