Track and Caption Any Motion: Query-Free Motion Discovery and Description in Videos

TL;DR

TCAM is a novel motion-centric framework that automatically discovers and describes multiple motion patterns in videos without user queries, leveraging motion dynamics and vision-language alignment for enhanced understanding.

Contribution

It introduces a unified approach combining motion discovery and description with contrastive vision-language training, enabling query-free multi-motion recognition and grounding.

Findings

Achieves 58.4% video-to-text retrieval on MeViS benchmark.

Attains 64.9 JF for spatial grounding accuracy.

Discovers an average of 4.8 relevant expressions per video with 84.7% precision.

Abstract

We propose Track and Caption Any Motion (TCAM), a motion-centric framework for automatic video understanding that discovers and describes motion patterns without user queries. Understanding videos in challenging conditions like occlusion, camouflage, or rapid movement often depends more on motion dynamics than static appearance. TCAM autonomously observes a video, identifies multiple motion activities, and spatially grounds each natural language description to its corresponding trajectory through a motion-field attention mechanism. Our key insight is that motion patterns, when aligned with contrastive vision-language representations, provide powerful semantic signals for recognizing and describing actions. Through unified training that combines global video-text alignment with fine-grained spatial correspondence, TCAM enables query-free discovery of multiple motion expressions via multi-head cross-attention. On the MeViS benchmark, TCAM achieves 58.4% video-to-text retrieval, 64.9 JF for spatial grounding, and discovers 4.8 relevant expressions per video with 84.7% precision, demonstrating strong cross-task generalization.