FIP: Endowing Robust Motion Capture on Daily Garment by Fusing Flex and Inertial Sensors

TL;DR

FIP is a novel system that uses flexible garments with sensors and advanced modeling to accurately capture human motion despite sensor displacements, enabling applications in Metaverse, rehabilitation, and fitness.

Contribution

The paper introduces a new motion capture system with a displacement-aware model and multimodal sensor fusion, significantly improving accuracy over existing IMU-based methods.

Findings

Achieves 19.5% reduction in angular error

Improves elbow angular error by 26.4%

Reduces positional error by 30.1%

Abstract

What if our clothes could capture our body motion accurately? This paper introduces Flexible Inertial Poser (FIP), a novel motion-capturing system using daily garments with two elbow-attached flex sensors and four Inertial Measurement Units (IMUs). To address the inevitable sensor displacements in loose wearables which degrade joint tracking accuracy significantly, we identify the distinct characteristics of the flex and inertial sensor displacements and develop a Displacement Latent Diffusion Model and a Physics-informed Calibrator to compensate for sensor displacements based on such observations, resulting in a substantial improvement in motion capture accuracy. We also introduce a Pose Fusion Predictor to enhance multimodal sensor fusion. Extensive experiments demonstrate that our method achieves robust performance across varying body shapes and motions, significantly outperforming SOTA IMU approaches with a 19.5% improvement in angular error, a 26.4% improvement in elbow angular error, and a 30.1% improvement in positional error. FIP opens up opportunities for ubiquitous human-computer interactions and diverse interactive applications such as Metaverse, rehabilitation, and fitness analysis.