TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis

TL;DR

TMR introduces a contrastive learning approach for text-to-3D human motion retrieval, significantly improving performance over previous methods by structuring the cross-modal latent space effectively.

Contribution

The paper extends the TEMOS model with contrastive loss for standalone text-to-motion retrieval and introduces a new benchmark for evaluation.

Findings

Outperforms prior methods with median rank reduced from 54 to 19

Effective cross-modal latent space structuring improves retrieval accuracy

Demonstrates potential for moment retrieval applications

Abstract

In this paper, we present TMR, a simple yet effective approach for text to 3D human motion retrieval. While previous work has only treated retrieval as a proxy evaluation metric, we tackle it as a standalone task. Our method extends the state-of-the-art text-to-motion synthesis model TEMOS, and incorporates a contrastive loss to better structure the cross-modal latent space. We show that maintaining the motion generation loss, along with the contrastive training, is crucial to obtain good performance. We introduce a benchmark for evaluation and provide an in-depth analysis by reporting results on several protocols. Our extensive experiments on the KIT-ML and HumanML3D datasets show that TMR outperforms the prior work by a significant margin, for example reducing the median rank from 54 to 19. Finally, we showcase the potential of our approach on moment retrieval. Our code and models are publicly available at https://mathis.petrovich.fr/tmr.