Lang2Motion: Bridging Language and Motion through Joint Embedding Spaces

TL;DR

Lang2Motion introduces a joint embedding framework that aligns language and motion representations, enabling accurate text-guided trajectory generation and transfer across diverse motion domains with improved retrieval and recognition performance.

Contribution

It is the first to align language and motion in a joint embedding space for explicit trajectory generation across arbitrary objects using real-world video data.

Findings

Achieves 34.2% Recall@1 in text-to-trajectory retrieval.

Outperforms video-based methods by 12.5 points in retrieval accuracy.

Improves motion accuracy by 33-52% over baselines.

Abstract

We present Lang2Motion, a framework for language-guided point trajectory generation by aligning motion manifolds with joint embedding spaces. Unlike prior work focusing on human motion or video synthesis, we generate explicit trajectories for arbitrary objects using motion extracted from real-world videos via point tracking. Our transformer-based auto-encoder learns trajectory representations through dual supervision: textual motion descriptions and rendered trajectory visualizations, both mapped through CLIP's frozen encoders. Lang2Motion achieves 34.2% Recall@1 on text-to-trajectory retrieval, outperforming video-based methods by 12.5 points, and improves motion accuracy by 33-52% (12.4 ADE vs 18.3-25.3) compared to video generation baselines. We demonstrate 88.3% Top-1 accuracy on human action recognition despite training only on diverse object motions, showing effective transfer across motion domains. Lang2Motion supports style transfer, semantic interpolation, and latent-space editing through CLIP-aligned trajectory representations.