Pose-Guided Residual Refinement for Interpretable Text-to-Motion Generation and Editing

TL;DR

This paper introduces PGR$^2$M, a hybrid pose-guided residual refinement method for text-to-motion generation and editing that enhances fidelity and control by combining interpretable pose codes with residuals learned via residual vector quantization.

Contribution

The paper proposes a novel hybrid representation combining pose codes and residuals for improved interpretability and detail in text-to-motion tasks, along with a pose-guided residual tokenizer and refinement framework.

Findings

PGR$^2$M outperforms baselines in Fréchet distance and reconstruction metrics.

It enables more accurate and detailed motion generation and editing.

User studies confirm its intuitive and structure-preserving editing capabilities.

Abstract

Text-based 3D motion generation aims to automatically synthesize diverse motions from natural-language descriptions to extend user creativity, whereas motion editing modifies an existing motion sequence in response to text while preserving its overall structure. Pose-code-based frameworks such as CoMo map quantifiable pose attributes into discrete pose codes that support interpretable motion control, but their frame-wise representation struggles to capture subtle temporal dynamics and high-frequency details, often degrading reconstruction fidelity and local controllability. To address this limitation, we introduce pose-guided residual refinement for motion (PGR$^2$M), a hybrid representation that augments interpretable pose codes with residual codes learned via residual vector quantization (RVQ). A pose-guided RVQ tokenizer decomposes motion into pose latents that encode coarse global structure and residual latents that model fine-grained temporal variations. Residual dropout further discourages over-reliance on residuals, preserving the semantic alignment and editability of the pose codes. On top of this tokenizer, a base Transformer autoregressively predicts pose codes from text, and a refine Transformer predicts residual codes conditioned on text, pose codes, and quantization stage. Experiments on HumanML3D and KIT-ML show that PGR$^2$M improves Fr\'echet inception distance and reconstruction metrics for both generation and editing compared with CoMo and recent diffusion- and tokenization-based baselines, while user studies confirm that it enables intuitive, structure-preserving motion edits.