EgoHDM: An Online Egocentric-Inertial Human Motion Capture, Localization, and Dense Mapping System

TL;DR

EgoHDM is an innovative online system that combines inertial sensors and a head-mounted camera to achieve real-time human motion capture, dense scene mapping, and accurate localization, even in complex terrains.

Contribution

It introduces the first dense scene mapping in real-time for egocentric human motion capture using inertial and visual data integration.

Findings

Reduces localization error by 41%

Improves camera pose accuracy by 71%

Enhances dense mapping accuracy by 46%

Abstract

We present EgoHDM, an online egocentric-inertial human motion capture (mocap), localization, and dense mapping system. Our system uses 6 inertial measurement units (IMUs) and a commodity head-mounted RGB camera. EgoHDM is the first human mocap system that offers dense scene mapping in near real-time. Further, it is fast and robust to initialize and fully closes the loop between physically plausible map-aware global human motion estimation and mocap-aware 3D scene reconstruction. Our key idea is integrating camera localization and mapping information with inertial human motion capture bidirectionally in our system. To achieve this, we design a tightly coupled mocap-aware dense bundle adjustment and physics-based body pose correction module leveraging a local body-centric elevation map. The latter introduces a novel terrain-aware contact PD controller, which enables characters to physically contact the given local elevation map thereby reducing human floating or penetration. We demonstrate the performance of our system on established synthetic and real-world benchmarks. The results show that our method reduces human localization, camera pose, and mapping accuracy error by 41%, 71%, 46%, respectively, compared to the state of the art. Our qualitative evaluations on newly captured data further demonstrate that EgoHDM can cover challenging scenarios in non-flat terrain including stepping over stairs and outdoor scenes in the wild.