FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions

TL;DR

FG-MDM introduces a novel zero-shot human motion generation framework that leverages ChatGPT-refined descriptions and a part-token diffusion model to produce diverse motions beyond training data.

Contribution

The paper presents a divide-and-conquer approach using large language models for fine-grained descriptions and a transformer-based diffusion model for zero-shot motion generation, which is a new methodology.

Findings

FG-MDM outperforms previous methods in zero-shot settings.

It generates diverse human motions beyond original dataset scope.

Fine-grained textual annotations improve motion generation quality.

Abstract

Recently, significant progress has been made in text-based motion generation, enabling the generation of diverse and high-quality human motions that conform to textual descriptions. However, generating motions beyond the distribution of original datasets remains challenging, i.e., zero-shot generation. By adopting a divide-and-conquer strategy, we propose a new framework named Fine-Grained Human Motion Diffusion Model (FG-MDM) for zero-shot human motion generation. Specifically, we first parse previous vague textual annotations into fine-grained descriptions of different body parts by leveraging a large language model. We then use these fine-grained descriptions to guide a transformer-based diffusion model, which further adopts a design of part tokens. FG-MDM can generate human motions beyond the scope of original datasets owing to descriptions that are closer to motion essence. Our experimental results demonstrate the superiority of FG-MDM over previous methods in zero-shot settings. We will release our fine-grained textual annotations for HumanML3D and KIT.