MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment

TL;DR

MoBind introduces a hierarchical contrastive learning framework that effectively aligns IMU signals with video-based skeletal motion, enabling precise cross-modal retrieval, synchronization, and action recognition.

Contribution

It presents a novel multi-level contrastive approach for fine-grained IMU-video alignment, decomposing full-body motion into local parts for improved semantic and temporal accuracy.

Findings

Outperforms baselines on multiple datasets

Achieves robust fine-grained temporal alignment

Preserves semantic consistency across modalities

Abstract

We aim to learn a joint representation between inertial measurement unit (IMU) signals and 2D pose sequences extracted from video, enabling accurate cross-modal retrieval, temporal synchronization, subject and body-part localization, and action recognition. To this end, we introduce MoBind, a hierarchical contrastive learning framework designed to address three challenges: (1) filtering out irrelevant visual background, (2) modeling structured multi-sensor IMU configurations, and (3) achieving fine-grained, sub-second temporal alignment. To isolate motion-relevant cues, MoBind aligns IMU signals with skeletal motion sequences rather than raw pixels. We further decompose full-body motion into local body-part trajectories, pairing each with its corresponding IMU to enable semantically grounded multi-sensor alignment. To capture detailed temporal correspondence, MoBind employs a hierarchical contrastive strategy that first aligns token-level temporal segments, then fuses local (body-part) alignment with global (body-wide) motion aggregation. Evaluated on mRi, TotalCapture, and EgoHumans, MoBind consistently outperforms strong baselines across all four tasks, demonstrating robust fine-grained temporal alignment while preserving coarse semantic consistency across modalities. Code is available at https://github.com/bbvisual/ MoBind.