Real-time RGBD-based Extended Body Pose Estimation

TL;DR

This paper introduces a real-time system for 3D human pose estimation using RGBD data, leveraging a parametric human mesh model and custom datasets, achieving high accuracy and 30 FPS performance.

Contribution

It presents a novel real-time RGBD-based human pose estimation system utilizing a parametric model and custom datasets, outperforming RGB-only methods and matching slower RGB-D solutions.

Findings

Outperforms state-of-the-art RGB-only methods in accuracy.

Achieves real-time 30 FPS performance on a single GPU.

Uses custom annotated datasets for training estimators.

Abstract

We present a system for real-time RGBD-based estimation of 3D human pose. We use parametric 3D deformable human mesh model (SMPL-X) as a representation and focus on the real-time estimation of parameters for the body pose, hands pose and facial expression from Kinect Azure RGB-D camera. We train estimators of body pose and facial expression parameters. Both estimators use previously published landmark extractors as input and custom annotated datasets for supervision, while hand pose is estimated directly by a previously published method. We combine the predictions of those estimators into a temporally-smooth human pose. We train the facial expression extractor on a large talking face dataset, which we annotate with facial expression parameters. For the body pose we collect and annotate a dataset of 56 people captured from a rig of 5 Kinect Azure RGB-D cameras and use it together with a large motion capture AMASS dataset. Our RGB-D body pose model outperforms the state-of-the-art RGB-only methods and works on the same level of accuracy compared to a slower RGB-D optimization-based solution. The combined system runs at 30 FPS on a server with a single GPU. The code will be available at https://saic-violet.github.io/rgbd-kinect-pose