Deep Multitask Architecture for Integrated 2D and 3D Human Sensing

TL;DR

This paper introduces a deep multitask neural network architecture that simultaneously performs 2D and 3D human pose recognition and reconstruction from monocular images, leveraging multi-dataset training for improved accuracy.

Contribution

The novel architecture enables joint training of recognition and reconstruction tasks using diverse datasets, improving robustness and accuracy in 2D and 3D human sensing from monocular images.

Findings

Achieves state-of-the-art results on multiple datasets.

Performs comparably to RGB-D systems in real-world scenarios.

Supports joint training across datasets with different annotations.

Abstract

We propose a deep multitask architecture for \emph{fully automatic 2d and 3d human sensing} (DMHS), including \emph{recognition and reconstruction}, in \emph{monocular images}. The system computes the figure-ground segmentation, semantically identifies the human body parts at pixel level, and estimates the 2d and 3d pose of the person. The model supports the joint training of all components by means of multi-task losses where early processing stages recursively feed into advanced ones for increasingly complex calculations, accuracy and robustness. The design allows us to tie a complete training protocol, by taking advantage of multiple datasets that would otherwise restrictively cover only some of the model components: complex 2d image data with no body part labeling and without associated 3d ground truth, or complex 3d data with limited 2d background variability. In detailed experiments based on several challenging 2d and 3d datasets (LSP, HumanEva, Human3.6M), we evaluate the sub-structures of the model, the effect of various types of training data in the multitask loss, and demonstrate that state-of-the-art results can be achieved at all processing levels. We also show that in the wild our monocular RGB architecture is perceptually competitive to a state-of-the art (commercial) Kinect system based on RGB-D data.

Tables

Table 1. Table 1: Body part labeling results for the Human80K test set . We report the performance of our network, trained on Human80K, LSP and both datasets jointly. Note that for LSP, the network was pre-trained on Human80K, as it otherwise failed to converge trained on LSP alone. We also compare with [ 15 ] , where comparisons are possible only for accuracies computed on the person foreground as the model of [ 15 ] does not predict the background label. Our model is able to predict the background class, thus we report the performance for the entire image (including the background class) as well as performance inside the silhouette (for classes associated to human body parts). Note that the best performance on Human80K is obtained with the network trained on Human80K, perhaps due to the noisy annotations added to LSP, and the naturally more similar training and testing distributions of Human80K.

Metric	DMHS${}_{\text{B}}$ - Human80K	DMHS${}_{\text{B}}$ - LSP (ft)	DMHS${}_{\text{B}}$ - Human80K & LSP	[15]
Avg. Acc. (%) per pixel (fg)	79.00	53.31	75.84	73.99
Avg. Acc. (%) per pixel (fg + bg)	91.15	83.30	89.92	-
Avg. Acc. (%) per class (fg)	67.35	43.40	64.83	-
Avg. Acc. (%) per class (fg + bg)	68.56	45.61	66.13	53.10

Table 2. Table 2: Body part labeling results for the LSP dataset . All models are initialized on the Human80K dataset, as networks trained only using LSP failed to converge. In this case the models trained jointly on Human80K and LSP produced the best results. This shows the importance of having accurate body part labeling annotations, obtained in simple imaging scenarios but for complex body poses, in combination with less accurate annotations but with more complex foreground-background appearance variations.

Metric	DMHS${}_{\text{B}}$ - Human80K	DMHS${}_{\text{B}}$ - LSP (ft)	DMHS${}_{\text{B}}$ - Human80K & LSP
Avg. Acc. (%) per pixel (fg)	50.52	60.54	61.16
Avg. Acc. (%) per pixel (fg + bg)	85.58	91.08	91.09
Avg. Acc. (%) per class (fg)	36.46	44.73	45.91
Avg. Acc. (%) per class (fg + bg)	38.77	46.88	48.01

Table 3. Table 3: 3d mean joint position error on the HumanEva-I dataset , computed between our predicted joints and the ground truth after an alignment with a rigid transformation. Comparisons with other competing methods show that DMHS achieves state-of-the-art performance.

Method	Walking			Avg.	Jog			Avg.	Box			Avg.	All
[39]	65.1	48.6	73.5	62.4	74.2	46.6	32.2	51.0	-	-	-	-	-
[4]	45.4	28.3	62.3	45.33	55.1	43.2	37.4	45.2	42.5	64.0	69.3	58.6	49.7
[48]	35.8	32.4	41.6	36.6	46.6	41.4	35.4	41.1	-	-	-	-	-
[42]	37.5	25.1	49.2	37.3	-	-	-	-	50.5	61.7	57.5	56.6	-
DMHS${}_{\text{R}}$(J,B,D)	27.1	18.4	39.5	28.3	37.6	28.9	27.6	31.4	30.5	45.8	48.0	41.5	33.7

Table 4. Table 4: 3d mean joint position error on the Human80K dataset . Different components of our model are compared to [ 15 ] .

Model	Avg. MPJPE (mm)
[15]	92.00
DMHS${}_{\text{R}}$(J)	128.05
DMHS${}_{\text{R}}$(D)	77.56
DMHS${}_{\text{R}}$(J,B)	118.68
DMHS${}_{\text{R}}$(J,D)	72.00
DMHS${}_{\text{R}}$(J,B,D)	63.35

Equations

$$L_J^t=\sum _{k=1}^{N_J}\sum _{z\in \mathcal{Z}}{\Vert J^t(z,k)-J^{\star }(z,k)\Vert }_2^2$$

$$L_B^t=-\frac{1}{|\mathcal{Z}|}\sum _{z\in \mathcal{Z}}\log (B_{z,B_z^{\star }}^t)$$

$$L_R^t=\sum _{i=1}^{N_R}\sqrt{\sum _{j=1}^3(f(R^t,i,j)-R^{\star }(i,j){)}^2+{ϵ}^2}$$

$$L=\sum _{t=1}^T(L_J^t+L_B^t+L_R^t)$$

Full text

1 Introduction

The visual analysis of humans has a broad spectrum of applications as diverse as autonomous vehicles, robotics, human-computer interaction, virtual reality, and digital libraries, among others. The problem is challenging due to the large variety of human poses and body proportions, occlusion, and the diversity of scenes, angles of observation, and backgrounds humans are pictured against. The monocular case, which is central and intrinsic in many scenarios like the analysis of photographs or video available on the web, adds complexity as depth information is missing for 3d reconstruction. This leads to geometric ambiguity and occlusion which are difficult to palliate compared to situations where multiple cameras are present.

A detailed analysis at both 2d and 3d levels, further exposes the need for both measurement and prior-knowledge, and the necessary inter-play between segmentation, reconstruction, and recognition within models that can jointly perform all tasks.

As training is essential, a major difficulty is the limited coverage of current datasets: 2d repositories like LSP [18] or MPI-II [3] exhibit challenging backgrounds, human body proportions, clothing, and poses, but offer single views, carry only approximate 2d joint location ground truth, and do not carry human segmentation or body part labeling information. Their size is also relatively small by today’s deep learning standards. In contrast, 3d datasets like HumanEva [38] or Human3.6M [16] offer extremely accurate 2d and 3d anatomical joint or body surface reconstructions and a variety of poses captured under multiple viewpoints. Human 3.6M is also large-scale. However, being captured indoors, the 3d datasets typically lack the background and clothing variability that represent a strength of the 2d datasets captured in the wild.111The situation is slightly more nuanced as some of the 3d datasets (e.g. Human3.6M) come with mixed-reality training setups where a moderately realistic graphics character was placed, in a geometrically correct setup, into a real scene and animated using human motion capture – arguably, though, a complete, fully realistic 2d and 3d training setting is still elusive. An open question is how one can leverage the separate strengths of existing 2d and 3d datasets towards training a model that can operate in realistic images and can offer accurate recognition and reconstruction estimates.

In this paper we propose one such deep learning model which, given a monocular RGB image, is able to fully automatically sense the humans at multiple levels of detail: figure-ground segmentation, body-part labeling at pixel level, as well as 2d and 3d pose estimation. By designing multi-task loss functions at different, recursive processing stages (human body joint detection and 2d pose estimation, semantic body part segmentation, 3d reconstruction) we are able to tie complete, realistic training scenarios by taking advantage of multiple datasets that would otherwise restrictively cover only some of the model component training (complex 2d image data with no body part labeling and without associated 3d ground truth, or complex 3d data with limited 2d background variability), leading to covariate shift and a lack of model expressiveness. In extensive experiments, including ablation studies performed using representative 2d and 3d datasets like LSP, HumanEva, or Human3.6M, we illustrate the model and show that state-of-the-art results can be achieved for both semantic body part segmentation and for 3d pose estimation.

2 Related Work

This work relates to 2d and 3d monocular human pose estimation methods as well as semantic segmentation using fully-trainable deep processing architectures. As prior-work is comprehensive in each sub-domain [35], we will here mostly cover some of the key techniques directly relating to our approach, with an emphasis towards deep architectures and methodologies aiming to integrate the different levels of 2d and 3d processing.

The problem of 2d human pose estimation has been approached initially using the pictorial structures and deformable part models where the kinematic tree structure of the human body offers a natural decomposition [11, 19, 7, 47, 29, 12]. In recent years, the deep learning methodology had a great impact over the state-of-the-art pose estimation models where different hierarchical feature extraction architectures have been combined with spatial constraints between the human body parts [44, 10, 43, 20, 30, 28, 23, 8, 13, 9, 41, 27, 24]. Recent deep architectures are obtained by cascading processing stages with similar structure but different parameters, where the system combines outputs of early layers with new information extracted directly from the image using learnt feature extractors. Such recursive schemes for 2d pose estimation appear in the work of [32, 46, 9] whereas a similar idea of feeding-back 3d estimates into 2d inference layers appears in [15]. Note that the iterative processing frameworks of [32, 15] are not deep, built on parts-based graphical models and random forests, respectively.

There is a vast literature on monocular 3d human pose estimation, including the analysis of kinematic ambiguities associated with the 3d pose [40, 5], as well as generative and discriminative methods for learning and inference [34, 36, 4, 1, 26, 37, 14, 30]. More recently, deep convolutional architectures have been employed in order to estimate 3d pose directly from images[21, 22, 33, 49] mostly in connection with 3d human motion capture datasets like HumanEva, Human3.6M, where the poses are challenging but the backgrounds are relatively simple. There is also interest in combining 2d and 3d estimation methods in order to obtain models capable of multiple task, and able to operate in realistic imaging conditions [31, 45, 2]. The most recent methods, [6, 33], rely on an a-priori 3d model that is fitted to anatomical body joint data obtained from an initial 2d pose estimate produced by a deep processing architecture, like the one of [13] or [43]. The methods rely on a state-of-the-art discriminative person detection and 2d pose estimation and introduce a generative fitting component to search in a space of admissible body proportion variations. Both approaches fit a statistical body shape and kinematic model to data, using one or multiple views and the joint assignment constraints from 2d human pose estimation. These methods use the 3d to 2d anatomical landmark assignment provided by [6, 33] as initialization for 3d human pose estimation. While this is effective, as shown in several challenging evaluation scenarios, the initial 3d shape and kinematic configuration still needs to be initialized manually or set to a key initial pose. This is in principle still prone to local optima as the monocular 3d human pose estimation cost is non-linear and non-convex even under perfect 3d to 2d model-image assignments [40].

We share with [15, 6, 33] the interest in building models that integrate 2d and 3d reasoning. We propose a fully trainable discriminative model for human recognition and reconstruction at 2d and 3d levels. We do not estimate human body shape, but we do estimate figure-ground segmentation, the semantic segmentation of the human body parts, as well as the 2d and 3d pose. The system is trainable, end-to-end, by means of multitask losses that can leverage the complementary properties of existing 2d and 3d human datasets. The model is fully automatic in the sense that both the human detection and body part segmentation and the 2d and 3d estimates are the result of recurrent stages of processing in a homogeneous, easy to understand and computationally efficient architecture. Our approach is complementary to [6, 33]: our model can benefit from a final optimization-based refinement and it would be useful to estimate the human body shape. In contrast, [6, 33] can benefit from the semantic segmentation of the human body parts for their shape fitting, and could use the accurate fully automatic 2d and 3d pose estimates we produce as initialization for their 3d to 2d refinement process.

3 Methodology

In this section we present our multitask multistage architecture. The idea of using multiple stages of recurrent feed-forward processing is inspired by architectures like [32, 46, 9] which focus separately on the 2d and 3d domains. However, we propose an uniform architecture for joint 2d and 3d processing that no prior method covers. Our choice of multi-task loss also makes it possible to exploit the complementary advantages of different datasets.

Conceptually, each stage of processing in our model produces recognition and reconstruction estimates and is constrained by specific training losses. Specifically, each stage $t$ is split into semantic processing $S^{t}$, and 3d reconstruction, $R^{t}$ (see fig. 1). At the same time, the semantic module $S^{t}$ is divided in two sub-tasks, one focusing on 2d pose estimation, $J^{t}$, and the other on body part labeling and figure-ground segmentation, $B^{t}$ (see fig. 2). The first one (i.e. $J^{t}$) feeds into the second one (i.e. $B^{t}$), while the semantic stages feed into the reconstruction stages. Each task consists of a total of six recurrent stages which take as input the image, the results of the previous stages of the same type (except for the first one), as well as inputs from other stages (2d pose estimation feeding into semantic body part segmentation and both feeding into 3d pose reconstruction). The inputs to each stage are individually processed and fused via convolutional networks in order to produce the corresponding outputs.

3.1 2D Human Body Joint Detection

The 2d pose estimation task is based on a recurrent convolutional architecture similar to [46]. Given an RGB image $I\in\mathbb{R}^{w\times h\times 3}$, we seek to correctly predict the locations of $N_{J}$ anatomically defined human body joints $p_{k}\in\mathcal{Z}\subset\mathbb{R}^{2}$, with $k\in{\{1\dots N_{J}}\}$. At each stage $t\in{\{1\dots T}\}$, where $T$ is the total number of stages, the network outputs belief maps $J^{t}\in\mathbb{R}^{w\times h\times N_{J}}$. The first stage of processing operates only on image evidence (a set of seven convolution and three pooling layers producing features ${\bf x}$) but for subsequent stages, the network also considers the information in the belief maps fed from the previous stage $J^{t-1}$ with a slightly different image feature function ${\bf x^{\prime}}$, defined as a set of four convolution and three pooling layers as in [46]. These features are transformed through a classification function $c^{t}_{J}$ to predict the body joint belief maps $J^{t}$. The function $c^{t}_{J}$ consists of a series of five convolutional layers, the first three of the form $(11\times 11\times 128)$, followed by a $(1\times 1\times 128)$ convolution and a final $(1\times 1\times N_{J})$ convolution that outputs $J^{t}$. The loss function at each stage $L^{t}_{J}$ minimizes the squared Euclidean distance between the predicted and ground truth belief maps, $J^{t}$ and $J^{\star}$ :

[TABLE]

In practice, this component of the model can be trained with data from both 2d and 3d datasets like LSP, where the ground truth is obtained manually, or HumanEva and Human3.6M where the ground truth is obtained automatically based on anatomical markers.

3.2 Semantic Body Part Segmentation

In semantic body part segmentation (body part labeling) we assign each image location $(u,v)\in\mathcal{Z}\subset\mathbb{R}^{2}$ one of $N_{B}$ anatomical body part labels (including an additional label for background), $b_{l}$, where $l\in{\{1\dots N_{B}}\}$. At each stage $t$, the network predicts, for each pixel location, the probability of each body part being present, $B^{t}\in\mathbb{R}^{w\times h\times N_{B}}$. Differently from the previous task, our aim now is to classify each pixel location, not only to identify the body joints. The loss function used changes from a squared Euclidean loss to a multinomial logistic loss:

[TABLE]

where $B^{\star}_{z}$ is the ground truth label for each image location $z=(u,v)$.

During the first stage of processing, we use convolutional representations based on the image (a series of convolution and pooling layers ${\bf x}$ with parameters tied from §3.1) and the 2d pose belief maps $J^{1}$ in order to predict the current body labels $B^{1}$. For each of the following stages, we also use the information present in the body labels at the previous stage, $B^{t-1}$, and rely on a series of four convolutional layers $c^{t}_{B}$ that learn to combine inputs obtained by stacking image features ${\bf x}$ and $B^{t-1}$. The function $c^{t}_{B}$ shares the same structure as the first four convolutions in $c^{t}_{J}$, but a classifier in the form of a $(1\times 1\times N_{B})$ convolution is applied after the fusion with the current 2d pose belief maps $J^{t}$, in order to obtain semantic probability maps $B^{t}$. An overview of our architecture together with the main dependencies is given in figure 3. Finally, we use an additional deconvolution layer [25] of size $16\times 16\times N_{B}$, such that the loss can be computed at the full resolution of the input image $I$.

In practice, realistic data for training this component of the loss is not as easy to obtain as 2d body joint positions. Human3.6M offers such training data, but we are also able to generate it approximately for LSP (see §4).

3.3 3D Pose Reconstruction

This model component is designed for the stage-wise, recurrent reconstruction of 3d human body configurations represented as set of $N_{R}$ 3d skeleton joints, from a single monocular image $I$. The estimate is obtained from the internal representations $R^{t}$. The 3d reconstruction module leverages information provided by the 2d semantic components $S^{t}$, incorporating the joint and body part labeling feature maps $J^{t}$ and $B^{t}$. Additionally, we insert a trainable function $c^{t}_{D}$, defined similarly to $c^{t}_{B}$, over image features, in order to obtain body reconstruction feature maps $D^{t}$. The module follows a similar flow as the previous ones: it reuses estimates at earlier processing stages, $R^{t-1}$, together with $S^{t}$ and $D^{t}$, in order to predict the reconstruction feature maps $R^{t}$. The processing stages and dependencies of this module are shown in fig. 4.

Procedurally, we first fuse $S^{t}$ and $D^{t}$, then apply a series of one $(3\times 3\times 128)$ convolution, one $(3\times 3\times 64)$ convolution, one $(1\times 1\times 64)$ convolution, followed by a pooling layer $(3\times 3)$ and a $(1\times 1\times 16)$ convolution. The output is concatenated with $R^{t-1}$ and convolved by a $(1\times 1\times 16)$ kernel that learns to combine the two components, producing the estimate $R^{t}$. The feature maps are then transformed to the desired dimensionality of the 3d human body skeleton by means of a fully connected layer. The loss $L^{t}_{R}$ is expressed as the mean per joint position error (MPJPE):

[TABLE]

where $R^{\star}$ are the 3d ground truth human joint positions, $f(\cdot)$ is the fully connected layer applied to $R^{t}$ and $\epsilon$ is a small constant that makes the loss differentiable.

This loss component can be trained with data from HumanEva and Human3.6M, but not from LSP or other 2d datasets as these lack 3d ground truth information. Although the backgrounds in HumanEva and Human3.6M are not as challenging as those in LSP, the use of a multitask loss makes the complete 2d and 3d model competitive not only in a laboratory setting but also in the wild (see §4 and fig. 5).

3.4 Integrated Multi-task Multi-stage Loss

Given the information provided in the previous sections, we are now able to define the complete multitask, multistage loss function of the model as follows:

[TABLE]

The loss allows us to conveniently train all the model component parameters, for different tasks, based on datasets where the annotations are challenging, or where annotations are missing entirely, as datasets with different coverage contribute to various components of the loss. Whenever we train using datasets with partial coverage, we could freeze the model components for which we do not have ground-truth. We can also simultaneously train all parameters, using datasets with partial and complete coverage: those examples for which we have ground truth at all levels will contribute to each of the loss components, whereas examples for which we have partial ground truth will only contribute to their corresponding losses.

4 Experiments

In order to evaluate our method, we use $3$ well-known datasets, the Leeds Sports Dataset (LSP) [18], HumanEva [38] and Human3.6M [16].

The LSP dataset consists of 2d pose annotated RGB images depicting sports people (athletics, badminton, baseball, gymnastics, parkour, soccer, tennis, volleyball). We employ both the original release containing $1,000$ training images and $1,000$ testing images, as well as the extended training release containing an additional $10,000$ images.

We use the first release of the HumanEva(-I) dataset. This dataset was captured by an accurate 3d motion capture system in a laboratory environment. There are six actions performed in total by three different subjects. As standard procedure [39, 4, 48, 42], we train our model on the train set and report results on the validation set, where we only consider every $5^{th}$ frame of the sequences walking, jog and box for all three subjects and a single frontal camera view.

Human80K is an 80,000 sample subset of the much larger 3.6 million human pose dataset Human3.6M [16]. The dataset is captured in a laboratory environment with a motion capture setup, and contains daily activities and interaction scenarios (providing directions, discussion, eating, activities while seating, greeting, taking photo, posing, making purchases, smoking, waiting, walking, sitting on chair, talking on the phone, walking dog, walking together). The actions are performed by $11$ actors, and captured by $4$ RGB cameras. The dataset is fully annotated and it contains RGB data, 2d body part labeling ground truth masks as well as accurate 2d and 3d pose reconstructions. Human80K consists of $55,144$ training and $24,416$ testing samples from Human3.6M. The samples from each original capture were selected such that the distance between each-other, in 3d space, is no more than 100 mm.

We use the Caffe [17] framework to train and test our architecture. The complete recognition and reconstruction pipeline takes approximately 400 ms per frame, in testing, on an Nvidia TITAN X (Pascal) 12GB GPU. We evaluate the recognition (2d body part labeling) and 3d reconstruction capabilities of our architecture. We use $T=6$ stages in our architecture for each sub-task component model (joint detection, semantic segmentation, 3d reconstruction) and report results only for the final stage of each sub-task, as it is the best performing according to validation.

4.1 Body Part Labeling

In order to evaluate the 2d body part labeling task, we use the Human80K and LSP datasets. We introduce additional annotations for LSP, as they are not available with the original release of the dataset which only provides the 2d anatomical joints. We create human body part annotations for LSP by using the annotated 2d skeleton joints and the kinematic tree. We produce circles for skeleton joints and ellipses for the individual body parts. We set the major axis to the size of the segment between the corresponding joints and estimate a constant value, for each body part, for the minor axis. The resulting masks (filtered by visual inspection) are of lower quality than those available for Human3.6M. The reason for augmenting LSP, is to enhance the variability in human appearance, body proportions and backgrounds in Human80K. Thus, we want to leverage the good quality labeling and pose variation of Human80K with the diverse appearance variations found in LSP.

For evaluation on Human80K, we compare with the results of [15] which represent the state-of-the-art for this task on this dataset. The authors of [15] assume that the silhouette of the person (the figure-ground segmentation) is given, and perform the body part labeling only on the foreground mask as an inference problem over a total of $24$ unique labels. Different from them, we do not make this assumption and consider the background as an extra class, thus building a model that predicts $25$ classes.

To extend the evaluation on LSP, we consider multiple scenarios: (a) training on Human80K with (b) fine-tuning on LSP and (c) training our architecture on LSP and Human80K simultaneously and test on both Human80K and LSP. In our setup, training using only LSP was not feasible, as in multiple experiments our network failed to converge. The labeling parameters (stages $B$) are initialized randomly, while the parameters corresponding to the 2d joint detection components, $J$, are initialized with the values of the network presented in [46], trained on MPI-II and LSP.

The performance of our body part labeling models for the Human80K and LSP is given in tables 1 and 2, respectively. We use the same evaluation metrics as in [15], i.e. average accuracy for the pixels contained within the ground truth silhouette and class normalized average accuracy. Basically, these two metrics apply only to the foreground classes. Additionally, we compute for all pixels, background and foreground, the average accuracy as well as the class normalized average accuracy.

From table 1 (Human80K testing), it can be noted that even though we solve a harder problem (by additionally estimating the figure-ground segmentation), the class normalized average precision greatly improves, with more than 10% over [15] for the models trained on (a) Human80K and (c) Human80K and LSP jointly. However, the model (b) initialized with parameters trained on Human80K, but fine-tuned on LSP, seems to have a performance drop caused by the low quality of the LSP body label annotations. In this scenario, as expected, the best performance is obtained by the model trained on Human80K, perhaps due to the fact that the test set distribution is better captured by the unaltered Human80K training set. This is not the case when testing on LSP, as it can be seen from table 2. The best performance is obtained by the model trained jointly on Human80K and LSP. This model is able to combine the pose variability and part labeling annotation quality of Human80K with the background and appearance variability of LSP, making it adequate for both laboratory settings and for images of people captured against challenging backgrounds.

The proposed models for the 2d body part labeling task are trained using an initial learning rate set to $10^{-10}$ and reduced at every $5$ epochs by a constant factor $\gamma=0.33$. During the learning process, the training data is augmented by randomly rotating with angles between $[-40^{\circ},+40^{\circ}]$, scaling by a factor in the range $[0.5,1.2]$ and horizontally flipping, in order to improve the diversity of the training set. Qualitative results of the semantic body part segmentation in challenging images are shown in fig. 5.

4.2 3D Human Pose Reconstruction

For the evaluation of the 3d pose reconstruction task, we use HumanEva-I and Human3.6M (specifically, Human80K), as they are both challenging datasets, containing complementary poses, and offering accurate 3d pose annotations. In all experiments, we use the same 3d evaluation metrics as other methods we compare against, and according to the standard practice. For the following experiments, note that our 2d semantic module components are trained on data from LSP and Human80K, whereas the 3d component of the module was pre-trained using Human80K.

We report results on the Human80K test set and investigate the impact of each of the input features $J$, $B$ and $D$ on the overall performance of the 3d reconstruction task (see table 4). Notice that models that use the ‘$D$-pipeline’ for the 3d reconstruction part, would correspond to convolutional networks that feed-forward from the image features, but do not leverage the semantic segmentation of the human body parts.

We observe that our fully integrated system, DMHS${}_{\mbox{R}}$(J,B,D), achieves the lowest error of $63.35$ mm. Besides the interest in computing additional detailed semantic human representations, it can be seen that by feeding in the results from other tasks – the body joint belief maps $J$ and the body labeling probability maps $B$ – the error is reduced considerably, from $77.56$ mm (for a model based on feed-forward processing and infering additional 2d joint position information) to $63.35$ mm. Notice also our significant gains with respect to the previous state-of-the-art results on Human80K, as reported by [15].

For HumanEva-I, we use the DMHS${}_{\mbox{R}}$(J,B,D) architecture trained on Human80K and fine-tuned for a few epochs ($6$) on the HumanEva-I training data. We use a subset of the training set containing $4,637$ samples. The fine-tuning step on HumanEva-I is performed in order to compensate for the differences in marker positioning with respect to Human80K, and in order to account for the different pose distributions w.r.t Human80K. In table 3, we compare our results against several state-of-the-art methods. We follow the standard evaluation procedure in [39, 4, 48, 42] and sample data from the validation set for walking, jogging and boxing activities. We use a single camera. We obtain considerable performance gains with respect to the previous state-of-the-art methods on HumanEva, even though we only use a small subset of the available training set.

For this model, we use an initial learning rate set to $10^{-7}$ and reduce it at every $5$ epochs by a constant factor $\gamma=0.66$. Qualitative results of our method can be seen in fig. 5 and fig. 6.

5 Conclusions

We have proposed a deep multitask architecture for fully automatic 2d and 3d human sensing (DMHS), including recognition and reconstruction, based on monocular images. Our system estimates the figure-ground segmentation and detects the human body joints, semantically identifies the body parts, and reconstructs the 2d and 3d pose of the person. The design of recurrent multi-task loss functions at multiple stages of processing supports the principled combination of the strengths of different 2d and 3d datasets, without being limited by their different weaknesses. In experiments we perform ablation studies, evaluate the effect of various types of training data in the multitask loss, and demonstrate that state-of-the-art-results can be achieved at all processing levels. We show that, even in the wild, our monocular RGB architecture is perceptually competitive to state-of-the art commercial RGB-D systems.

6 Acknowledgment

This work was supported in part by CNCS-UEFISCDI under PCE-2011-3-0438, JRP-RO-FR-2014-16.