Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidance

TL;DR

This paper introduces EAIL, a framework that uses egocentric action cues from head-mounted IMU signals, combined with vision-language guidance, to improve inertial localization within 3D point clouds, addressing challenges of noise and diverse human motions.

Contribution

The paper proposes a novel multimodal learning approach that correlates IMU-based action cues with environmental features for improved localization and action recognition.

Findings

Outperforms state-of-the-art inertial localization methods.

Effectively correlates human actions with environmental structures.

Enables concurrent action recognition from IMU and visual data.

Abstract

This paper presents a novel inertial localization framework named Egocentric Action-aware Inertial Localization (EAIL), which leverages egocentric action cues from head-mounted IMU signals to localize the target individual within a 3D point cloud. Human inertial localization is challenging due to IMU sensor noise that causes trajectory drift over time. The diversity of human actions further complicates IMU signal processing by introducing various motion patterns. Nevertheless, we observe that some actions captured by the head-mounted IMU correlate with spatial environmental structures (e.g., bending down to look inside an oven, washing dishes next to a sink), thereby serving as spatial anchors to compensate for the localization drift. The proposed EAIL framework learns such correlations via hierarchical multi-modal alignment with vision-language guidance. By assuming that the 3D point cloud of the environment is available, it contrastively learns modality encoders that align short-term egocentric action cues in IMU signals with local environmental features in the point cloud. The learning process is enhanced using concurrently collected vision and language signals to improve multimodal alignment. The learned encoders are then used in reasoning the IMU data and the point cloud over time and space to perform inertial localization. Interestingly, these encoders can further be utilized to recognize the corresponding sequence of actions as a by-product. Extensive experiments demonstrate the effectiveness of the proposed framework over state-of-the-art inertial localization and inertial action recognition baselines.