LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning

TL;DR

LaMP introduces a novel language-motion pretraining framework that significantly improves motion generation, retrieval, and captioning by creating a more aligned and informative language-motion latent space, surpassing previous CLIP-based methods.

Contribution

This work presents LaMP, a new pretraining model that transitions from static image-text embeddings to a dynamic language-motion space, enhancing task performance in motion-related applications.

Findings

Substantial improvements in motion generation, retrieval, and captioning tasks.

Introduction of LaMP-BertScore for better motion-text alignment evaluation.

Effective motion-informative text embeddings enhance relevance and semantics.

Abstract

Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. The code of our method will be made public.