DiffCap: Diffusion-based Real-time Human Motion Capture using Sparse IMUs and a Monocular Camera

TL;DR

This paper introduces DiffCap, a diffusion-based framework that fuses sparse IMU data and monocular camera inputs for real-time human motion capture, handling occlusions and view changes effectively.

Contribution

It presents a novel diffusion-based approach that seamlessly integrates visual and inertial signals for robust, real-time human pose estimation.

Findings

Achieves state-of-the-art pose estimation accuracy.

Effectively handles occlusions and missing visual data.

Demonstrates robustness and real-time performance.

Abstract

Combining sparse IMUs and a monocular camera is a new promising setting to perform real-time human motion capture. This paper proposes a diffusion-based solution to learn human motion priors and fuse the two modalities of signals together seamlessly in a unified framework. By delicately considering the characteristics of the two signals, the sequential visual information is considered as a whole and transformed into a condition embedding, while the inertial measurement is concatenated with the noisy body pose frame by frame to construct a sequential input for the diffusion model. Firstly, we observe that the visual information may be unavailable in some frames due to occlusions or subjects moving out of the camera view. Thus incorporating the sequential visual features as a whole to get a single feature embedding is robust to the occasional degenerations of visual information in those frames. On the other hand, the IMU measurements are robust to occlusions and always stable when signal transmission has no problem. So incorporating them frame-wisely could better explore the temporal information for the system. Experiments have demonstrated the effectiveness of the system design and its state-of-the-art performance in pose estimation compared with the previous works. Our codes are available for research at https://shaohua-pan.github.io/diffcap-page.