VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera

TL;DR

This paper introduces a real-time, monocular RGB camera-based method for capturing full 3D human skeletal poses with temporal stability, suitable for applications like character control, outperforming previous RGB-D approaches in versatility.

Contribution

It presents the first real-time monocular RGB method combining a CNN-based pose regressor with kinematic skeleton fitting for stable 3D human pose estimation.

Findings

Achieves real-time 3D pose estimation comparable to offline methods.

Works effectively in outdoor scenes and with low-quality cameras.

Outperforms RGB-D methods in applicability and sometimes accuracy.

Abstract

We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fully-convolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control---thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e. it works for outdoor scenes, community videos, and low quality commodity RGB cameras.

Tables

Table 1. Table 1 . Comparison of our network against state of the art on MPI-INF-3DHP test set, using ground-truth bounding boxes. We report the Percentage of Correct Keypoints measure in 3D, and the Area Under the Curve for the same, as proposed by MPI-INF-3DHP. We additionally report the Mean Per Joint Position Error in mm. Higher PCK and AUC is better, and lower MPJPE is better.

		Stand/		Sit On	Crouch/	On the
Network	Scales	Walk	Exercise	Chair	Reach	Floor	Sports	Misc.	Total
		PCK	PCK	PCK	PCK	PCK	PCK	PCK	PCK	AUC	MPJPE(mm)
Ours (ResNet 100)	0.7, 1.0	87.6	76.4	71.4	71.6	47.8	82.5	78.9	75.0	39.5	127.8
	1.0	86.4	72.3	68.0	65.4	40.7	80.5	76.3	71.4	36.9	142.8
Ours (ResNet 50)	0.7, 1.0	87.7	77.4	74.7	72.9	51.3	83.3	80.1	76.6	40.4	124.7
	1.0	86.7	73.9	69.8	66.1	44.7	82.0	79.4	73.3	37.8	138.7
[Mehta et al., 2016] (ResNet 100)	0.7, 1.0	86.6	75.3	74.8	73.7	52.2	82.1	77.5	75.7	39.3	117.6
	1.0	86.3	72.4	71.5	67.6	49.2	81.0	76.2	73.2	37.8	126.6

Table 2. Table 2 . Results on MPI-INF-3DHP test set with the bounding box corners randomly jittered between +/- 40px to emulate noise from a BB estimator. Our fully-convolutional formulation is more robust than a comparative fully-connected formulation. The evaluation is at a single scale (1.0).

	Stand/		Sit On	Crouch/	On the
Network	Walk	Exerc.	Chair	Reach	Floor	Sport	Misc.	Total
	PCK	PCK	PCK	PCK	PCK	PCK	PCK	PCK	AUC
Ours (ResNet 100)	86.0	71.0	65.0	61.1	37.4	78.9	75.5	69.5	35.8
Ours (ResNet 50)	84.9	69.4	65.1	61.9	40.8	78.6	77.6	70.1	35.7
[Mehta et al., 2016]	81.2	64.2	67.1	62.1	43.5	76.0	71.1	67.8	34.0

Table 3. Table 3 . Results of our raw CNN predictions on Human3.6m, evaluated on the ground truth bounding box crops for all frames of Subject 9 and 11. Our CNNs use only Human3.6m as the 3D training set, and are pretrained for 2D pose prediction. The error measure used is Mean Per Joint Position Error (MPJPE) in millimeters. Note again that the error measure used is not robust, and subject to obfuscation from occasional large mispredictions, such as those exhibited by our raw CNN predictions.

									Sit		Take			Walk	Walk
Method	Direct	Discuss	Eating	Greet	Phone	Posing	Purch.	Sitting	Down	Smoke	Photo	Wait	Walk	Dog	Pair	All
[Zhou et al., 2015b]	87.4	109.3	87.1	103.2	116.2	106.9	99.8	124.5	199.2	107.4	143.3	118.1	79.4	114.2	97.7	113.0
[Tekin et al., 2016c]	102.4	147.7	88.8	125.3	118.0	112.3	129.2	138.9	224.9	118.4	182.7	138.8	55.1	126.3	65.8	125.0
[Yu et al., 2016]	85.1	112.7	104.9	122.1	139.1	105.9	166.2	117.5	226.9	120.0	135.9	117.7	137.4	99.3	106.5	126.5
[Ionescu et al., 2014b]	132.7	183.6	132.4	164.4	162.1	150.6	171.3	151.6	243.0	162.1	205.9	170.7	96.6	177.1	127.9	162.1
[Zhou et al., 2016]	91.8	102.4	97.0	98.8	113.4	90.0	93.8	132.2	159.0	106.9	125.2	94.4	79.0	126.0	99.0	107.3
[Pavlakos et al., 2016]	58.6	64.6	63.7	62.4	66.9	57.7	62.5	76.8	103.5	65.7	70.7	61.6	69.0	56.4	59.5	66.9
[Mehta et al., 2016]	52.6	63.8	55.4	62.3	71.8	52.6	72.2	86.2	120.6	66.0	79.8	64.0	48.9	76.8	53.7	68.6
[Tekin et al., 2016b]	85.0	108.8	84.4	98.9	119.4	98.5	93.8	73.8	170.4	85.1	95.7	116.9	62.1	113.7	94.8	100.1
Ours (ResNet 100)	61.7	77.8	64.6	70.3	90.5	61.9	79.8	113.2	153.1	80.9	94.4	75.1	54.9	83.5	61.0	82.5
Ours (ResNet 50)	62.6	78.1	63.4	72.5	88.3	63.1	74.8	106.6	138.7	78.8	93.8	73.9	55.8	82.0	59.6	80.5

Table 4. Table 4 . Results on MPI-INF-3DHP test set with the 3D joint position lookup in the location-maps done using the ground truth 2D locations rather the predicted 2D locations. We see that the location maps have captured better 3D pose information, which can perhaps be extracted through optimization methods operating directly on heatmaps and location-maps. The evaluation uses 2 scales (0.7, 1.0).

	Stand/		Sit On	Crouch/	On the
Network	Walk	Exerc.	Chair	Reach	Floor	Sport	Misc.	Total
	PCK	PCK	PCK	PCK	PCK	PCK	PCK	PCK	AUC
Ours (ResNet 100)	88.1	80.9	74.0	76.1	56.3	82.9	80.2	77.8	41.0
Ours (ResNet 50)	88.0	81.8	78.6	77.4	59.3	82.8	81.2	79.4	41.6
[Mehta et al., 2016]	86.6	75.3	74.8	73.7	52.2	82.1	77.5	75.7	39.3

Equations

$$\text{Loss}(x_j)=\Vert {\mathbf{H}}_j^{\text{GT}}\odot ({\mathbf{X}}_j-{\mathbf{X}}_j^{\text{GT}}){\Vert }_2\text{,}$$

$${\mathbf{BL}}_j=\sqrt[]{{\mathbf{\Delta X}}_j\odot {\mathbf{\Delta X}}_j+{\mathbf{\Delta Y}}_j\odot {\mathbf{\Delta Y}}_j+{\mathbf{\Delta Z}}_j\odot {\mathbf{\Delta Z}}_j}\text{.}$$

$$E_{\text{total}}(\mathbf{\theta },\mathbf{d})$$

$$+E_{\text{smooth}}(\mathbf{\theta },\mathbf{d})+E_{\text{depth}}(\mathbf{\theta },\mathbf{d}),$$

$$E_{\text{proj}}=\Vert \Pi ({\mathbf{P}}_t^{\text{G}})-{\mathbf{K}}_t{\Vert }_2\text{ and }{E}_{\text{IK}}=\Vert ({\mathbf{P}}_t^{\text{G}}-\mathbf{d})-{\mathbf{P}}_t^{\text{L}}{\Vert }_2\text{,}$$

Full text

This work is was funded by the ERC Starting Grant project CapReal (335545). Dan Casas was supported by a Marie Curie Individual Fellow, grant agreement 707326. 

 

1. Introduction

Optical skeletal motion capture of humans is widely used in applications such as character animation for movies and games, sports and biomechanics, and medicine. To overcome the usability constraints imposed by commercial systems requiring marker suits [Menache, 2000], researchers developed marker-less motion capture methods that estimate motion in more general scenes using multi-view video [Moeslund et al., 2006], with recent solutions being real-time [Stoll et al., 2011]. The swell in popularity of applications such as real-time motion-driven 3D game character control, self-immersion in 3D virtual and augmented reality, and human–computer interaction, has led to new real-time full-body motion estimation techniques using only a single, easy to install, depth camera, such as the Microsoft Kinect [Microsoft Corporation, 2010, 2013, 2015]. RGB-D cameras provide valuable depth data which greatly simplifies monocular pose reconstruction. However, RGB-D cameras often fail in general outdoor scenes (due to sunlight interference), are bulkier, have higher power consumption, have lower resolution and limited range, and are not as widely and cheaply available as color cameras.

Skeletal pose estimation from a single color camera is a much more challenging and severely underconstrained problem. Monocular RGB body pose estimation in 2D has been widely researched, but estimates only the 2D skeletal pose [Wei et al., 2016; Felzenszwalb and Huttenlocher, 2005; Felzenszwalb et al., 2010; Bourdev and Malik, 2009; Ferrari et al., 2009; Pishchulin et al., 2013]. Learning-based discriminative methods, in particular deep learning methods [Insafutdinov et al., 2016; Newell et al., 2016; Tompson et al., 2014; Lifshitz et al., 2016], represent the current state of the art in 2D pose estimation, with some of these methods demonstrating real-time performance [Wei et al., 2016; Cao et al., 2016]. Monocular RGB estimation of the 3D skeletal pose is a much harder challenge tackled by relatively fewer methods [Tekin et al., 2016c; Tekin et al., 2016b; Zhou et al., 2016, 2015; Zhou et al., 2015b; Bogo et al., 2016]. Unfortunately, these methods are typically offline, and they often reconstruct 3D joint positions individually per image, which are temporally unstable, and do not enforce constant bone lengths. Most approaches also capture local 3D pose relative to a bounding box, and not the full global 3D pose. This makes them unsuitable for applications such as real-time 3D character control.

In this paper, we present the first method that captures temporally consistent global 3D human pose—in terms of joint angles of a single, stable kinematic skeleton—in real-time (30 Hz) from a single RGB video in a general environment. Our approach builds upon the top performing single RGB 3D pose estimation methods using convolutional neural networks (CNNs) [Pavlakos et al., 2016; Mehta et al., 2016]. High accuracy requires training comparably deep networks which are harder to run in real-time, partly due to additional preprocessing steps such as bounding box extraction. Mehta  et al. [2016] use a 100-layer architecture to predict 2D and 3D joint positions simultaneously, but is unsuitable for real-time execution. To improve runtime, we use a shallower 50-layer network. However, for best quality at real-time frame rates, we do not merely use a shallower variant, but extend it to a novel fully-convolutional formulation. This enables higher accuracy 2D and 3D pose regression, in particular of end effectors (hands, feet), in real-time. In contrast to existing solutions our approach allows operation on non-cropped images, and where run-time is a concern, it can be used to bootstrap a simple bounding box tracker. We also combine the CNN-based joint position regression with an efficient optimization step to fit a 3D skeleton to these reconstructions in a temporally stable way, yielding the global pose and joint angles of the skeleton. In summary, we contribute by proposing the first real-time method to capture global 3D kinematic skeleton pose from single RGB video. To strike a good compromise between computational complexity and accuracy, our method combines:

• •

A new real-time, fully-convolutional 3D body pose formulation using CNNs that yields 2D and 3D joint positions simultaneously and forgoes the need to perform expensive bounding box computations.

• •

Model-based kinematic skeleton fitting against the 2D/3D pose predictions to produce temporally stable joint angles of a metric global 3D skeleton, in real time.

Our real-time method achieves state-of-the-art accuracy comparable to the best offline RGB pose estimation methods on standard 3D human body pose benchmarks, particularly for end effector positions (Section 1). Our results are qualitatively comparable to, and sometimes better than, state-of-the-art single RGB-D methods [Girshick et al., 2011], even commercial ones [Microsoft Corporation, 2015]. We experimentally show that this makes ours the first single-RGB method usable for similar real-time 3D applications—so far only feasible with RGB-D input—such as game character control or immersive first person virtual reality (VR). We further show that our method succeeds in settings where existing RGB-D methods would not, such as outdoor scenes, community videos, and even with low quality video streams from ubiquitous mobile phone cameras.

2. Related Work

Our goal is stable 3D skeletal motion capture from (1) a single camera (2) in real-time. We focus the discussion of related work on approaches from the large body of marker-less motion capture research that contributed to attaining either of these properties.

Multi-view:  With multi-view setups markerless motion-captue solutions attain high accuracy. Tracking of a manually initialized actor model from frame to frame with a generative image formation model is common. See [Moeslund et al., 2006] for a complete overview. Most methods target high quality with offline computation [Bregler and Malik, 1998; Howe et al., 1999; Sidenbladh et al., 2000; Starck and Hilton, 2003; Loper and Black, 2014]. Real-time performance can be attained by representing the actor with Gaussians [Wren et al., 1997; Stoll et al., 2011; Rhodin et al., 2015] and other approximations [Ma and Wu, 2014], in addition to formulations allowing model-to-image fitting. However, these tracking-based approaches often lose track in local minima of the non-convex fitting functions they optimize and require separate initialization, e.g. using [Sminchisescu and Triggs, 2001; Bogo et al., 2016; Rhodin et al., 2016b]. Robustness could be increased with a combination of generative and discriminative estimation [Elhayek et al., 2016], even from a single input view [Sminchisescu et al., 2006; Rosales and Sclaroff, 2006], and egocentric perspective [Rhodin et al., 2016a]. We utilize generative tracking components to ensure temporal stability, but avoid model projection through a full image formation model to speed up estimation. Instead, we combine discriminative pose estimation with kinematic fitting to succeed in our underconstrained setting.

Monocular Depth-based:  The additional depth channel provided by RGB-D sensors has led to robust real-time pose estimation solutions [Baak et al., 2011; Ganapathi et al., 2012; Wei et al., 2012; Shotton et al., 2013; Ma and Wu, 2014; Ye and Yang, 2014] and the availability of low-cost devices has enabled a range of new applications. Even real-time tracking of general deforming objects [Zollhöfer et al., 2014] and template-free reconstruction [Newcombe et al., 2015; Innmann et al., 2016; Orts-Escolano et al., 2016; Dou et al., 2016] has been demonstrated. RGB-D information overcomes forward-backwards ambiguities in monocular pose estimation. Our goal is a video solution that overcomes depth ambiguities without relying on a specialized active sensor.

Monocular RGB:  Monocular generative motion capture has only been shown for short clips and when paired with strong motion priors [Urtasun et al., 2006] or in combination with discriminative re-initialization [Sminchisescu et al., 2006; Rosales and Sclaroff, 2006], since generative reconstruction is fundamentally underconstrained. Using photo-realistic template models for model fitting enables more robust monocular tracking of simple motions, but requires more expensive offline computation [de La Gorce et al., 2008]. Sampling-based methods avoid local minima [Deutscher and Reid, 2005; Balan et al., 2005; Gall et al., 2010; Bo and Sminchisescu, 2010]. However, real-time variants can not guarantee global convergence due to a limited number of samples, such as particle swarm optimization techniques [Oikonomidis et al., 2011]. Structure-from-motion techniques exploit motion cues in a batch of frames [Garg et al., 2013], and have also been applied to human motion estimation [Gotardo and Martinez, 2011; Lee et al., 2013; Park and Sheikh, 2011; Zhu et al., 2011]. However, batch optimization does not apply to our real-time setting, where frames are streamed sequentially. For some applications manual annotation and correction of frames is suitable, for instance to enable movie actor reshaping [Jain et al., 2010] and garment replacement in video [Rogge et al., 2014]. In combination with physical constraints, highly accurate reconstructions are possible from monocular video [Wei and Chai, 2010]. Vondrak  et al. [2012] succeed without manual annotation by simulating biped-controllers, but require batch-optimization. While these methods can yield high-quality reconstructions, interaction and expensive optimization preclude live applications.

Discriminative 2D human pose estimation is often an intermediate step to monocular 3D pose estimation. Pictorial structure approaches infer body part locations by message passing over a huge set of pose-states [Agarwal and Triggs, 2006; Felzenszwalb and Huttenlocher, 2005; Ferrari et al., 2009; Andriluka et al., 2009; Bourdev and Malik, 2009; Johnson and Everingham, 2010] and have been extended to 3D pose estimation [Balan et al., 2007; Sigal et al., 2012; Amin et al., 2013; Belagiannis et al., 2014]. Recent approaches outperform these methods in computation time and accuracy by leveraging large image databases with 2D joint location annotation, which enables high accuracy prediction with deep CNNs [Hu et al., 2016; Belagiannis and Zisserman, 2016; Pishchulin et al., 2016; Insafutdinov et al., 2016; Wei et al., 2016], on multiple GPUs, even at real-time rates [Cao et al., 2016]. Given 2D joint locations, lifting them to 3D pose is challenging. Existing approaches use bone length and depth ordering constraints [Taylor, 2000; Mori and Malik, 2006], sparsity assumptions [Wang et al., 2014; Zhou et al., 2015, 2015a], joint limits [Akhter and Black, 2015], inter-penetration constraints [Bogo et al., 2016], temporal dependencies [Rhodin et al., 2016b], and regression [Yasin et al., 2016]. Treating 3D pose as a hidden variable in 2D estimation is an alternative [Brau and Jiang, 2016]. However, the sparse set of 2D locations loses image evidence, e.g. on forward-backwards orientation of limbs, which leads to erroneous estimates in ambiguous cases. To overcome these ambiguities, discriminative methods have been proposed that learn implicit depth features for 3D pose directly from more expressive image representations. Rosales and Sclaroff regress 3D pose from silhouette images with the specialized mappings architecture [2000], Agarwal and Triggs with linear regression [2006], and Elgammal and Lee through a joint embedding of images and 3D pose [2004]. Sminchisescu further utilized temporal consistency to propagate pose probabilities with a Bayesian mixture of experts Markov model [2007]. Relying on the recent advances in machine learning techniques and compute capabilities, approaches for direct 3D pose regression from the input image have been proposed, using structured learning of latent pose [Tekin et al., 2016a; Li et al., 2015a], joint prediction of 2D and 3D pose [Li and Chan, 2014; Tekin et al., 2016b; Yasin et al., 2016], transfer of features from 2D datasets [Mehta et al., 2016], novel pose space formulations [Pavlakos et al., 2016] and classification over example poses [Pons-Moll et al., 2014; Rogez and Schmid, 2016]. Relative per-bone predictions [Li and Chan, 2014], kinematic skeleton models [Zhou et al., 2016], or root centered joint positions [Ionescu et al., 2014a] are used as the eventual output space. Such direct 3D pose regression methods capture depth relations well, but 3D estimates usually do not accurately match the true 2D location when re-projected to the image, because estimations are done in cropped images that lose camera perspective effects, using a canonical height, and minimize 3D loss instead of projection to 2D. Furthermore, they only deliver joint positions, are temporally unstable, and none has shown real-time performance. We propose a method to combine 2D and 3D estimates in real-time along with temporal tracking. It is inspired by the method of Tekin et al. [2016c], where batches of frames are processed offline after motion compensation, and is related to the recently proposed per-frame combination of 2D and 3D pose [Tekin et al., 2016b].

Notably, only few methods target real-time monocular reconstruction. Exceptions are the regression of 3D pose from Haar features by Bissacco et al. [2007] and detection of a set of discrete poses from edge direction histograms in the vicinity of the previous frame pose [Taycher et al., 2006]. Both only obtain temporally unstable, coarse pose, not directly usable in our applications. Chai and Hodgins obtain sufficient quality to drive virtual avatars in real-time, but require visual markers [Chai and Hodgins, 2005]. The use of CNNs in real time has been explored for variants of the object detection problem, for instance bounding box detection and pedestrian detection methods have leveraged application specific architectures [Liu et al., 2016; Redmon et al., 2015; Angelova et al., 2015] and preprocessing steps [Ren et al., 2015].

In a similar vein, we propose a 3D pose estimation approach that leverages a novel fully-convolutional CNN formulation to predict 2D and 3D pose jointly. In combination with inexpensive preprocessing and an optimization based skeletal fitting method, it enables high accuracy pose estimation, while running at more than 30 Hz.

3. Overview

Our system is capable of obtaining a temporally consistent, full 3D skeletal pose of a human from a monocular RGB camera. Estimating 3D pose from a single RGB camera is a challenging, underconstrained problem with inherent ambiguities. Figure 2 provides an overview of our method to tackle this challenging problem. It consists of two primary components. The first is a convolutional neural network (CNN) to regress 2D and 3D joint positions under the ill-posed monocular capture conditions. It is trained on annotated 3D human pose datasets [Ionescu et al., 2014b; Mehta et al., 2016], additionally leveraging annotated 2D human pose datasets [Andriluka et al., 2014; Johnson and Everingham, 2010] for improved in-the-wild performance. The second component combines the regressed joint positions with a kinematic skeleton fitting method to produce a temporally stable, camera-relative, full 3D skeletal pose.

CNN Pose Regression:  The core of our method is a CNN that predicts both 2D, and root (pelvis) relative 3D joint positions in real-time. The new proposed fully-convolutional pose formulation leads to results on par with the state-of-the-art offline methods in 3D joint position accuracy (see Section 1 for details). Being fully-convolutional, it can operate in the absence of tight crops around the subject. The CNN is capable of predicting joint positions for a diverse class of activities regardless of the scene settings, providing a strong basis for further pose refinement to produce temporally consistent full-3D pose parameters

Kinematic Skeleton Fitting:  The 2D and the 3D predictions from the CNN, together with the temporal history of the sequence, can be leveraged to obtain temporally consistent full-3D skeletal pose, with the skeletal root (pelvis) localized in camera space. Our approach uses an optimization function that: (1) combines the predicted 2D and 3D joint positions to fit a kinematic skeleton in a least squares sense, (2) ensures temporally smooth tracking over time. We further improve the stability of the tracked pose by applying filtering steps at different stages.

Skeleton Initialization (Optional):  The system is set up with a default skeleton which works well out of the box for most humans. For more accurate estimates, the relative body proportions of the underlying skeleton can be adapted to that of the subject, by averaging CNN predictions for a few frames at the beginning. Since monocular reconstruction is ambiguous without a scale reference, the CNN predicts height normalized 3D joint positions. Users only need to provide their height (distance from head to toe) once, so that we can track the 3D pose in true metric space.

4. Real-time Monocular 3D Pose Estimation

In this section, we describe in detail the different components of our method to estimate a temporally consistent 3D skeletal motion from monocular RGB input sequences. As input, we assume a continuous stream of monocular RGB images $\{...,\mathbf{I}_{t-1},\mathbf{I}_{t}\}$. For frame $t$ in the input stream, the final output of our approach is $\mathbf{P}^{\text{G}}_{t}$ which is the full global 3D skeletal pose of the person being tracked. Because this output is already temporally consistent and in global 3D space, it can be readily used in applications such as character control.

We use the following notation for the output in the intermediate components of our method. The CNN pose regressor jointly estimates the 2D joint positions $\mathbf{K}_{t}$ and root-relative 3D joint positions ${\mathbf{P}^{\text{L}}_{t}}$ (Section 4.1). The 3D skeleton fitting component combines the 2D and 3D joint position predictions to estimate a smooth, temporally consistent pose $\mathbf{P}^{\text{G}}_{t}(\bm{\theta},\mathbf{d})$, which is parameterized by the global position $\mathbf{d}$ in camera space, and joint angles $\bm{\theta}$ of the kinematic skeleton $S$. $J$ indicates the number of joints. We drop the frame-number subscript $t$ in certain sections to aid readability.

4.1. CNN Pose Regression

The goal of CNN pose regression is to obtain joint positions, both, in 2D image space and 3D. For 2D pose estimation with neural nets, the change of formulation from direct regression of $x,y$ body-joint coordinates [Toshev and Szegedy, 2014] to a heatmap based body-joint detection formulation [Tompson et al., 2014] has been the key driver behind the recent developments in 2D pose estimation. The heatmap based formulation naturally ties image evidence to pose estimation by predicting a confidence heatmap $\mathbf{H}_{j,t}$ over the image plane for each joint $j\in\{1..J\}$.

Existing approaches to 3D pose estimation lack such an image-to-prediction association, often directly regressing the root-relative joint locations [Ionescu et al., 2014a], leading to predicted poses whose extent of articulation doesn’t reflect that of the person in the image. See Figure 9. Treating pose as a vector of joint locations also causes a natural gravitation towards networks with fully-connected formulations [Tekin et al., 2016a; Mehta et al., 2016; Rogez and Schmid, 2016; Yu et al., 2016], restricting the inputs to tight crops at a fixed resolution, a limitation that needs to be overcome. These methods assume the availability of tight bounding boxes, which necessitates supplementation with separate bounding box estimators for actual usage, which further adds to the run-time of these methods. The fully-convolutional formulation of Pavlakos  et al. [Pavlakos et al., 2016] seeks to alleviate some of these issues, but is limited by the expensive per-joint volumetric formulation, which still relies on cropped input and does not scale well to larger image sizes.

We overcome these limitations through our new formulation, by extending the 2D heatmap formulation to 3D using three additional location-maps $\mathbf{X}_{j},\mathbf{Y}_{j},\mathbf{Z}_{j}$ per joint $j$, capturing the root-relative locations $x_{j}$, $y_{j}$ and $z_{j}$ respectively. To have the 3D pose prediction linked more strongly to the 2D appearance in the image, the $x_{j}$, $y_{j}$ and $z_{j}$ values are read off from their respective location-maps at the position of the maximum of the corresponding joint’s 2D heatmap $\mathbf{H}_{j}$, and stored in ${\mathbf{P}^{\text{L}}}$ = {$\mathbf{x}$, $\mathbf{y}$, $\mathbf{z}$}, where $\mathbf{x}\in\mathbb{R}^{1\times J}$ is a vector that stores the coordinate $x$ location of each joint maximum. The pose formulation is visualized in Figure 3. Networks using this fully-convolutional formulation are not constrained in input image size, and can work without tight crops. Additionally, the network provides 2D and 3D joint location estimates without additional overhead, which we exploit in subsequent steps for real-time estimation. Section 1 shows the improvements afforded by this formulation.

Loss Term:  To enforce the fact that we are only interested in $x_{j}$, $y_{j}$ and $z_{j}$ from their respective maps at joint $j$’s 2D location, the joint location-map loss is weighted stronger around the joint’s 2D location. We use the L2 loss. For $x_{j}$ is the loss formulation is

[TABLE]

where GT indicates ground truth and $\odot$ is the Hadamard product. The location maps are weighted with the respective ground truth 2D heatmap $\mathbf{H}^{\text{GT}}_{j}$, which in turn have confidence equal to a Gaussian with a small support localized at joint $j$’s 2D location. Note that no structure is imposed on the location-maps. The structure that emerges in the predicted location-maps is indicative of the correlation of $x_{j}$ and $y_{j}$ with root relative location of joint $j$ in the image plane. See Figure 3.

Network Details:  We use the proposed formulation to adapt the ResNet50 network architecture of He  et al. [2016]. We replace the layers of ResNet50 from res5a onwards with the architecture depicted in Figure 5, producing the heatmaps and location-maps for all joints $j\in\{1..J\}$. After training, the Batch Normalization [Ioffe and Szegedy, 2015] layers are merged with the weights of their preceding convolution layers to improve the speed of the forward pass.

Intermediate Supervision:  We predict the 2D heatmaps and 3D location-maps from the features at res4d and res5a, tapering down the weights of intermediate losses with increasing iteration count. Additionally, similar to the root-relative location-maps $\mathbf{X}_{j}$, $\mathbf{Y}_{j}$ and $\mathbf{Z}_{j}$, we predict kinematic parent-relative location-maps $\mathbf{\Delta X}_{j}$, $\mathbf{\Delta Y}_{j}$ and $\mathbf{\Delta Z}_{j}$ from the features at res5b and compute bone length-maps as:

[TABLE]

These intermediate predictions are subsequently concatenated with the intermediate features, to give the network an explicit notion of bone lengths to guide the predictions. See Figure 5.

Experiments showed that the deeper variants of ResNet offer only small gains for a substantial increase (1.5$\times$) in computation time, prompting us to choose ResNet50 to enable real-time, yet highly accurate joint location estimation with the proposed formulation.

Training:  The network is pretrained for 2D pose estimation on MPII [Andriluka et al., 2014] and LSP [Johnson and Everingham, 2010, 2011] to allow superior in-the-wild performance, as proposed by Mehta  et al. [Mehta et al., 2016]. For 3D pose, we use MPI-INF-3DHP [Mehta et al., 2016] and Human3.6m [Ionescu et al., 2014b]. We take training sequences for all subjects except S9 and S11 from Human3.6m. We sample frames as per [Ionescu et al., 2014a]. For MPI-INF-3DHP, we consider all 8 training subjects. We choose sequences from all 5 chest-high cameras, 2 head-high cameras (angled down) and 1 knee-high camera (angled up) to learn some degree of invariance to the camera viewpoint. The sampled frames have at least one joint move by $>200mm$ between them. We use various combinations of background, occluder (chair), upper-body clothing and lower-body clothing augmentation for  70% of the selected frames. We train with person centered crops, and use image scale augmentation at 2 scales ($0.7\times,1.0\times$), resulting in 75k training samples for Human3.6m and 100k training samples for MPI-INF-3DHP. Figure 4 shows a few representative frames of training data. In addition to the 17 joints typically considered, we use foot tip positions. The ground truth joint positions are with respect to a height normalized skeleton (knee–neck height 92 cm). We make use of the Caffe [2014] framework for training, and use the Adadelta [Zeiler, 2012] solver with learning rate tapered down with increasing iterations.

Bounding Box Tracker:  Existing offline solutions process each frame in a separate person-localization and bounding box (BB) cropping step [Tekin et al., 2016c; Mehta et al., 2016] or assume bounding boxes are available [Tekin et al., 2016a; Li et al., 2015b; Li and Chan, 2014; Pavlakos et al., 2016; Zhou et al., 2016]. Although our fully-convolutional formulation allows the CNN to work without requiring cropping, the run-time of the CNN is highly dependent on the input image size. Additionally, the CNN is trained for subject sizes in the range of 250–340 px in the frame, requiring averaging of predictions at multiple image scales per frame (scale space search) if processing the full frame at each time step. Guaranteeing real-time rates necessitates restricting the size of the input to the network and tracking the scale of the person in the image to avoid searching the scale space in each frame. We do this in an integrated way. The 2D pose predictions from the CNN at each frame are used to determine the BB for the next frame through a slightly larger box around the predictions. The smallest rectangle containing the keypoints $\mathbf{K}$ is computed and augmented with a buffer area $0.2\times$ the height vertically and $0.4\times$ the width horizontally. To stabilize the estimates, the BB is shifted horizontally to the centroid of the 2D predictions, and its corners are filtered with a weighted average with the previous frame’s BB using a momentum of $0.75$. To normalize scale, the BB crop is resized to 368x368 px. The BB tracker starts with (slow) multi-scale predictions on the full image for the first few frames, and hones in on the person in the image making use of the BB-agnostic predictions from the fully convolutional network. The BB tracking is easy to implement and without runtime overhead, since the proposed fully-convolutional network outputs 2D and 3D pose jointly and operates on arbitrary input sizes.

4.2. Kinematic Skeleton Fitting

Applying per-frame pose estimation techniques on a video does not exploit and ensure temporal consistency of motion, and small pose inaccuracies lead to temporal jitter, an unacceptable artifact for most graphics applications. We combine the 2D and 3D joint positions in a joint optimization framework, along with temporal filtering and smoothing, to obtain an accurate, temporally stable and robust result. First, the 2D predictions $\mathbf{K}_{t}$ are temporally filtered [Casiez et al., 2012] and used to obtain the 3D coordinates of each joint from the location-map predictions, giving us $\mathbf{P}^{\text{L}}_{t}$. To ensure skeletal stability, the bone lengths inherent to $\mathbf{P}^{\text{L}}_{t}$ are replaced by the bone lengths of the underlying skeleton in a simple retargeting step that preserves the bone directions of $\mathbf{P}^{\text{L}}_{t}$. The resulting 2D and 3D predictions are combined by minimizing the objective energy

[TABLE]

for skeletal joint angles $\bm{\theta}$ and the root joint’s location in camera space $\mathbf{d}$. The 3D inverse kinematics term $E_{\text{IK}}$ determines the overall pose by similarity to the 3D CNN output $\mathbf{P}^{\text{L}}_{t}$. The projection term $E_{\text{proj}}$ determines global position $\mathbf{d}$ and corrects the 3D pose by re-projection onto the detected 2D keypoits $\mathbf{K_{t}}$. Both terms are implemented with the L2 loss,

[TABLE]

where $\Pi$ is the projection function from 3D to the image plane, and $\mathbf{P}^{\text{G}}_{t}=\mathbf{P}^{\text{G}}_{t}(\bm{\theta},\mathbf{d})$. We assume the pinhole projection model. If the camera calibration is unknown a vertical field of view of 54 degrees is assumed. Temporal stability is enforced with smoothness prior $E_{\text{smooth}}=\|\widehat{\mathbf{P}^{\text{G}}_{t}}\|_{2}$, penalizing the acceleration $\widehat{\mathbf{P}^{\text{G}}_{t}}$. To counteract the strong depth uncertainty in monocular reconstruction, we penalize large variations in depth additionally with $E_{\text{depth}}=\|[\widetilde{\mathbf{P}^{\text{G}}_{t}}]_{z}\|_{2}$ where $[\widetilde{\mathbf{P}^{\text{G}}_{t}}]_{z}$ is the $z$ component of 3D velocity $\widetilde{\mathbf{P}^{\text{G}}_{t}}$. Finally, the 3D pose is also filtered with the 1 Euro filter [Casiez et al., 2012].

Parameters:  The energy terms $E_{\text{IK}},E_{\text{proj}},E_{\text{smooth}}$ and $E_{\text{depth}}$ are weighted with $\omega_{\text{IK}}=1,\omega_{\text{proj}}=44,\omega_{\text{smooth}}=0.07$ and $\omega_{\text{depth}}=0.11$, respectively. The parameters of the 1 Euro Filter [Casiez et al., 2012] are empirically set to $f_{c_{\text{min}}}=1.7$, $\beta=0.3$ for filtering $\mathbf{K}_{t}$, to $f_{c_{\text{min}}}=0.8$, $\beta=0.4$ for $\mathbf{P}^{\text{L}}_{t}$, and to $f_{c_{\text{min}}}=20$, $\beta=0.4$ for filtering $\mathbf{P}^{\text{G}}_{t}$. Our implementation uses the Levenberg-Marquardt algorithm from the Ceres library [Agarwal et al., 2017].

5. Results

We show live applications of our system at 30 Hz. The reconstruction quality is high and we demonstrate the usefulness of our method 3D character control, embodied virtual reality, and pose tracking from low quality smartphone camera streams. See Section 5.3 and Figure 1. Results are best observed in motion in the supplemental video. The importance of the steps towards enabling these applications with a video solution are thoroughly evaluated in more than 10 sequences. Results are comparable in quality to depth-camera based solutions like the Kinect [Microsoft Corporation, 2013] and significantly outperform existing monocular video-based solutions. As the qualitative baseline we choose the state-of-the-art 2D to 3D lifting approach of Zhou  et al.  [2015] and the 3D regression approach of Mehta  et al.  [2016], which estimate joint-positions offline. The accuracy improvements are further quantitatively validated on the established H3.6M dataset [Ionescu et al., 2014b] and the MPI-INF-3DHP dataset [Mehta et al., 2016]. The robustness to diverse persons, clothing and scenes is demonstrated on several real-time examples and community videos. Please see our project webpage for more results and details111http://gvv.mpi-inf.mpg.de/projects/VNect/.

Computations are performed on a 6-core Xeon CPU, 3.8 GHz and a single Titan X (Pascal architecture) GPU. The CNN computation takes $\approx$18 ms, the skeleton fitting $\approx$7–10 ms, and preprocessing and filtering 5 ms.

5.1. Comparison with Active Depth Sensors (Kinect)

We synchronously recorded video from an RGB camera and a co-located Kinect sensor in a living room scenario. Figure 6 shows representative frames. Although the depth sensor provides additional information, our reconstructions from just RGB are of a similar quality. The Kinect results are of comparable stability to ours, but yield erroneous reconstructions when limbs are close to scene objects, such as when sitting down. Our RGB solution, however, succeeds in this case, although is slightly less reliable in depth estimation. A challenging case for both methods is the tight crossing of legs. Please see the supplemental video for a visual comparison.

The video solution succeeds also in situations with direct sunlight (Figure 7), where IR-based depth cameras are inoperable. Moreover, RGB cameras can simply be equipped with large field-of-view (FOV) lenses and, despite strong distortions, successfully track humans [Rhodin et al., 2016a]. On the other hand, existing active sensors are limited to relatively small FOVs, which severely limits the tracking volume.

5.2. Comparison with Video Solutions

Qualitative Evaluation:  We qualitatively compare against the 3D pose regression method of Mehta  et al.  [2016] and Zhou  et al.  [2015] on Sequence 6 (outdoor) of MPI-INF-3DHP test set. Our results are comparable to the quality of these offline methods (see Figure 10). However, the per frame estimates of these offline methods exhibits jitter over time, a drawback of most existing solutions. Our full pose results are temporally stable and are computed at real-time frame rate of 30 Hz.

The kinematic skeleton fitting estimates global translation $\mathbf{d}$. Figure 8 demonstrates that estimates are drift-free, the feet position matches with the same reference point after performing a circular walk. The smoothness constraint in depth direction limits sliding of the character away from the character, as pictured in the supplemental video sequences.

Quantitative Evaluation: 

We compare our method with the state-of-the-art approach of Mehta  et al.  [2016] on the MPI-INF-3DHP dataset, using the more robust Percentage of Correct Keypoints metric (3D PCK @150mm) on the 14 joints spanned by head, neck, shoulders, elbow, wrist, hips, knees and ankles. We train both, our model, as well as that of Mehta  et al. on the same data (Human3.6m + MPI-INF-3DHP), as detailed in Section 4.1, to be compatible in terms of the camera viewpoints selected, and use ResNet100 as the base architecture for a fair comparison. Table 1 shows the results of the raw 3D predictions from our network on ground-truth bounding box cropped frames. We see that the results are comparable to that of Mehta et al. The slight increase in accuracy on going to a 50-layer network is possibly due to the better gradient estimates coming from larger mini-batches that can be fit into memory while training, on account of the smaller size of the network. Evidence that our method ties the estimated 3D positions strongly to image appearance than previous methods can also be gleaned from the fact that our approach performs significantly better for activity classes such as Standing/Walking, Sports and Miscellaneous without significant self-occlusions. We do lose some performance on activity classes with significant self-occlusion such as Sitting/Lying on the floor. We additionally report the Mean Per Joint Position Error (MPJPE) numbers in mm. Note that MPJPE is not a robust measure, and is heavily influenced by large outliers, and hence the worse performance on the MPJPE measure (124.7mm vs 117.6mm) despite the better 3D PCK results (76.6% vs 75.7%).

We further investigate the nature of errors of our method. We first look at the joint-wise breakup of accuracy of our fully-convolutional ResNet100 CNN predictions vs Mehta  et al. ’s formulation with fully-connected layers. Figure 11 shows that the accuracy of ankles for our formulation is significantly better, while the accuracy of the head is markedly worse.

In Figure 9, we visually compare the two methods, further demonstrating the strong tie-in to image appearance that our formulation affords, and the downsides of the strong tie-in. We also show that our method is prone to occasional large mispredictions when the body joint 2D location detector misfires. It is these large outliers that obfuscate the reported MPJPE numbers. Figure 12, which plots the fraction of mispredicted joints vs. the error threshold on MPI-INF-3DHP test set shows that our method has a higher fraction of per-joint mispredictions beyond 300mm. It explains the higher MPJPE numbers compared to Mehta  et al. despite equivalent PCK performance. The various filtering stages employed in the full pipeline ameliorate these large mispredictions.

For Human3.6m, we follow the protocol as in earlier work [Pavlakos et al., 2016; Tekin et al., 2016b; Tekin et al., 2016c], and evaluate on all actions and cameras for subject number 9 and 11, and report Mean Per Joint Position Error (mm) for root relative 3D joint positions from our network. See Table 3. Note that despite the occasional large outliers affecting the MPJPE measure, our predictions are still better than most of the existing methods.

The accuracy attained from single view methods is still below that of real-time multi-view methods, which can achieve a mean accuracy of the order of 10mm [Stoll et al., 2011].

Generalization to Different Persons and Scenes: 

We tested our method on a variety of actors, it succeeds for different body shapes, gender and skin tone. See supplemental video. To further validate the robustness we applied the methods to community videos from YouTube, see Figure 1. It generalizes well to the different backgrounds and camera types.

Model Components: 

To demonstrate that our fully-convolutional pose formulation is less sensitive to inexact cropping than networks using a fully-connected formulation, we emulate a noisy BB estimator by jittering the ground-truth bounding box corners of MPI-INF-3DHP test set uniformly at random in the range of +/- 40 px. This also captures scenarios where one or more end effectors are not in the frame, so a loss in accuracy is expected for all methods. Table 2 shows that the fully-connected formulation of Mehta  et al. suffers a worse hit in accuracy than our approach, going down by 7.9 PCK, while our comparable network goes down by only 5.5 PCK.

We show the effect of the various components of our full pipeline on the TS1 sequence of MPI-INF-3DHP test set in Figure 13. Without the $E_{\text{IK}}$ component of $E_{\text{total}}$ the tracking accuracy goes down to a PCK of 46.1% compared to a PCK of 81.7% when $E_{\text{IK}}$ is used. The raw CNN 3D predictions in conjunction with the BB tracker result in a PCK of 80.3%. Using $E_{\text{IK}}$ in $E_{\text{total}}$ produces consistently better results for all thresholds lower than 150 mm. This shows the improvements brought about by our skeleton fitting term. Additionally, as shown in the supplementary video, using 1 Euro filtering produces qualitatively better results, but the overall PCK decreases slightly (79.7%) due to slower recovery from tracking failures.

The influence of the smoothness and filtering steps on the temporal consistency are further analyzed in the supplemental video.

5.3. Applications

Our approach is suitable for various interactive applications since it is real-time, temporally stable, fully automatic, and exports data directly in a format amenable to 3D character control.

Character Control:  Real-time motion capture solutions provide a natural interface for game characters and virtual avatars, which go beyond classical mouse and gamepad control. We applied our method on motions common in activities like tennis, dance, and juggling, see Figures 1 and 14. The swing of the arm and leg motion is nicely captured and could, for instance, be used in a casual sports and dancing game, but also for motion analysis of professional athletes to optimize their motion patterns. We also show successful results in non front-facing motions such as turning and writing on a wall, as well as squatting.

Virtual Reality:  The recent availability of cheap head-mounted displays has sparked a range of new applications. Many products use handheld devices to track the user’s hand position for interaction. Our solution enables them from a single consumer color camera. Beyond interaction, our marker-less full-body solution enables embodied virtual reality, see Figure 1. A rich immersive feeling is created by posing a virtual avatar of the user exactly to their own real pose. With our solution the real and virtual pose are aligned such that users perceive the virtual body as their own.

Ubiquitous Motion Capture with Smartphones:  Real-time monocular 3D pose estimation lends itself to application on low quality smartphone video streams. By streaming the video to a machine with sufficient capabilities for our algorithm, one can turn any smartphone into a lightweight, fully-automatic, handheld motion capture sensor, see Figure 15 and the accompanying video. Since smartphones are widespread, it enables the aforementioned applications for casual users without requiring additional sensing devices.

6. Limitations

Depth estimation from a monocular image is severely ill posed, slight inaccuracies in the estimation can lead to largely different depth estimates, which manifests also in our results in slight temporal jitter. We claim improved stability and temporal consistency compared to existing monocular RGB 3D pose estimation methods. This uncertainty could be further reduced with domain specific knowledge, e.g., foot-contact constraints when the floor location is known, and head-pose stabilization with the position of head-mounted-displays in VR applications, which is readily obtained with IMU-sensors.

A downside of our CNN prediction formulation is that mispredictions of 2D joint locations result in implausible 3D poses. This is ameliorated in the tracker through skeleton retargeting and pose filtering. This could be addressed directly in the CNN through imposition of stronger interdependencies between predictions. Additionally, the performance on poses with significant amounts of self occlusion remains a challenge.

Further, very fast motions can exceed the convergence radius of our IK optimization, but the integration of per frame 2D and 3D pose yields quick recovery from erroneous poses. Initial experiments with $256\times 256$ px input to the CNN show that much higher frame rates are possible with no loss in accuracy.

7. Discussion

The availability of sufficient annotated 3D pose training data remains an issue. Even the most recent annotated real 3D pose data sets, or combined real/synthetic data sets [Chen et al., 2016; Ionescu et al., 2014b; Mehta et al., 2016] are a subset of real world human pose, shape, appearance and background distributions. Recent top performing methods explicitly address this data sparsity by training similarly deep networks, but with architectural changes enabling improved intermediate training supervision [Mehta et al., 2016].

Our implementation only supports a single person, although the proposed fully-convolutional formulation could be scaled to multiple persons. Such an extension is currently precluded due to the lack of multi-person datasets, required to train multi-person 3D pose regressors. One possible approach is to adapt the multi-person 2D pose methods of Insafutdinov  et al.  [2016] and Cao  et al.  [2016].

We also analyze the impact of 2D joint position mispredictions on the 3D joint position predictions from our fully-convolutional formulation. We decouple the 3D predictions from the 2D predictions by looking up the 3D joint positions from their location-maps using the ground truth 2D joint positions. See Table 4. We see a 3D PCK improvement of 2.8, which is congruent with the notion of a stronger tie-in of the predicted joint positions with the image plane, which causes the 3D joint predictions to be erroneous when 2D joint detection misfires. The upside of this is that the 3D predictions can be improved through improvements to 2D joint position prediction. Alternatively, optimization formulations that directly operate on the heatmaps and the location-maps could be constructed. Our fully-convolutional formulation can also benefit from iterative refinement, akin to heatmap-based 2D pose estimation approaches [Newell et al., 2016; Hu et al., 2016].

8. Conclusion

We have presented the first method that estimates the 3D kinematic pose of a human, including global position, in a stable, temporally consistent manner from a single RGB video stream at 30 Hz. Our approach combines a fully-convolutional CNN that regresses 2D and 3D joint positions and a kinematic skeleton fitting method, producing a real-time temporally stable 3D reconstruction of the motion. In contrast to most existing approaches, our network can operate without strict bounding boxes, and facilitates inexpensive bounding box tracking. We have shown results in a variety of challenging real-time scenarios, including live streaming from a smartphone camera, as well as in community videos. A number of applications have been presented, such as embodied VR and interactive character control for computer games.

Qualitative and quantitative evaluations demonstrate that our approach compares to offline state-of-the-art monocular RGB methods and approaches the quality of real-time RGB-D methods. Hence, we believe our method is a significant step forward to democratizing 3D human pose estimation, lifting both the need for special cameras such as the IR-based depth cameras, as well as the long and heavy processing times.