KETA: Kinematic-Phrases-Enhanced Text-to-Motion Generation via Fine-grained Alignment

TL;DR

KETA introduces a novel text-to-motion generation method that uses kinematic phrases as an intermediate representation, improving alignment and motion accuracy through fine-grained supervision and iterative refinement.

Contribution

This work presents KETA, a new approach that decomposes text into kinematic phrases and aligns them with motion segments, enhancing the quality and consistency of generated motions.

Findings

KETA achieves up to 1.19x better R precision.

KETA reduces FID values by up to 2.34x.

It outperforms existing T2M models in accuracy and quality.

Abstract

Motion synthesis plays a vital role in various fields of artificial intelligence. Among the various conditions of motion generation, text can describe motion details elaborately and is easy to acquire, making text-to-motion(T2M) generation important. State-of-the-art T2M techniques mainly leverage diffusion models to generate motions with text prompts as guidance, tackling the many-to-many nature of T2M tasks. However, existing T2M approaches face challenges, given the gap between the natural language domain and the physical domain, making it difficult to generate motions fully consistent with the texts. We leverage kinematic phrases(KP), an intermediate representation that bridges these two modalities, to solve this. Our proposed method, KETA, decomposes the given text into several decomposed texts via a language model. It trains an aligner to align decomposed texts with the KP segments extracted from the generated motions. Thus, it's possible to restrict the behaviors for diffusion-based T2M models. During the training stage, we deploy the text-KP alignment loss as an auxiliary goal to supervise the models. During the inference stage, we refine our generated motions for multiple rounds in our decoder structure, where we compute the text-KP distance as the guidance signal in each new round. Experiments demonstrate that KETA achieves up to 1.19x, 2.34x better R precision and FID value on both backbones of the base model, motion diffusion model. Compared to a wide range of T2M generation models. KETA achieves either the best or the second-best performance.