HumanMeshNet: Polygonal Mesh Recovery of Humans

TL;DR

HumanMeshNet is a novel model for 3D human body reconstruction from monocular images that learns mesh representations implicitly, ensuring smooth surface reconstruction and enabling real-time performance with high accuracy.

Contribution

It introduces a multi-branch, multi-task approach that regresses mesh vertices and incorporates mesh topology regularization, improving efficiency and accuracy over existing methods.

Findings

Achieves comparable accuracy to state-of-the-art methods

Operates with significantly reduced computational complexity

Demonstrates generalizability to hand mesh prediction

Abstract

3D Human Body Reconstruction from a monocular image is an important problem in computer vision with applications in virtual and augmented reality platforms, animation industry, en-commerce domain, etc. While several of the existing works formulate it as a volumetric or parametric learning with complex and indirect reliance on re-projections of the mesh, we would like to focus on implicitly learning the mesh representation. To that end, we propose a novel model, HumanMeshNet, that regresses a template mesh's vertices, as well as receives a regularization by the 3D skeletal locations in a multi-branch, multi-task setup. The image to mesh vertex regression is further regularized by the neighborhood constraint imposed by mesh topology ensuring smooth surface reconstruction. The proposed paradigm can theoretically learn local surface deformations induced by body shape variations and can therefore learn high-resolution meshes going ahead. We show comparable performance with SoA (in terms of surface and joint error) with far lesser computational complexity, modeling cost and therefore real-time reconstructions on three publicly available datasets. We also show the generalizability of the proposed paradigm for a similar task of predicting hand mesh models. Given these initial results, we would like to exploit the mesh topology in an explicit manner going ahead.

Tables

Table 1. Table 1 : Comparison with other methods on UP3D’s full test set  [ 13 ]

		Surface	Joints	PA. Surface	PA. Joint
Output	Method	Error	Error	Error	Error
P	Pavlakos et al. [19]	117.7	-	-	-
P	Lasner et al. [13]	169.8	-	-	-
P	NBF [18]	-	-	-	82.3
V	BodyNet [28]	80.1	-	-	-
S	Baseline	151.4	130.8	93.8	83.7
S	HMNet	130.4	112.5	77.6	69.6
S	HMNetOracle	60.3	51.5	42.9	37.9

Table 2. Table 2 : Comparison with state-of-the-art methods on SURREAL’s test set  [ 29 ]

		Surface	Joint
Output	Method	Error	Error
P	Tung et al. [27]	74.5	64.4
	Pavlakos et al. [19]	151.5	-
	SMPLR  [15]	75.4	55.8
V	BodyNet [28]	65.8	-
S	Baseline	101	85.7
	HMNet[subsampled]	86.9	72.4
	HMNet	86.6	71.9
	HMNetOracle	63.5	49.1

Table 3. Table 3 : Joint Reconstruction error as per Protocol 2 of Bogo et al.   [ 4 ] on Human 3.6M  [ 10 ] . Refer to Section  4.1 for details on 3D mesh supervision

3D mesh		PA. Joint
Supervision	Method	Error
No	Ramakrishnan et al. [20]	157.3
	Zhou et al. [35]	106.7
	SMPLify  [4]	82.3
	SMPLify 91 kps  [13]	80.7
	Pavlakos et al. [19]	75.9
	HMR  [11]	56.8
	HMNet(Ours)	60.9
Yes	NBF  [18]	59.9
	SMPLR  [15]	56.4
	CMR  [12]	50.1

Table 4. Table 4 : Effect of each network module on the reconstruction error on UP-3D dataset. SM DP and SM GT denotes segmentation obtained from Densepose and groundtruth respectively.

Config.	Input	PA. Surface	PA. Joint
		Error	Error
Baseline	RGB	93.8	83.7
Single Task	SMDP	82.9	74.6
Single Task	RGB+SMDP	79.2	71.04
HMNet	RGB+SMDP	77.6	69.6
HMNetOracle	RGB+SMGT	42.9	37.9

Table 5. Table 5 : Overview of the run time (in Frames Per Second, FPS) of various algorithms. Numbers have been picked up from the respective papers. All methods have used 1080Ti or equivalent GPU.

Method	Output	FPS
SMPLify [3]	P	0.01
SMPLify, 91 kps [13]		0.008
Decision Forests [13]		7.69
HMR [11]	P	25
Pavlakos [19]		20
Direct Prediction [13]		2.65
Baseline	S	175.4
HMNet		28.01
HMNetOracle		173.17
Fusion4D [7]	S	31

Equations

$$L=L_S+({\lambda }_1\ast L_J)+({\lambda }_2\ast L_{JS})$$

$$L_S=\sum _{\forall V_i}||V_i-\widehat{V_i}|{|}_2$$

$$V_i=\frac{1}{|N(V_i)|}\sum _{V_j\in N(V_i)}{V}_j\forall V_i$$

$$L_J=\sum _{\forall J_i}||J_i-\widehat{J_i}|{|}_2$$

$$L_{JS}=\sum _{\forall J_i}||J_i-J{S}_i|{|}_2$$

Full text

1 Introduction

Recovering a 3D human body shape from a monocular image is an ill posed problem in computer vision with great practical importance for many applications, including virtual and augmented reality platforms, animation industry, e-commerce domain, etc.

Some of the recent deep learning methods employ volumetric regression to recover the voxel grid reconstruction of human body models from a monocular image [30, 28]. Although volumetric regression enables recovering a more accurate surface reconstruction, they do so without an animatable skeleton [30], which limits their applicability for some of the aforementioned applications. [28] attempted to overcome this limitation by fitting a parametric body model on the volumetric reconstruction using a sillhoute reprojection loss. Nevertheless, in general, such methods yield reconstructions of low resolution at higher computational cost (regression over the cubic voxel grid) and often suffer from broken or partial body parts.

Alternatively, parametric body model [2, 14, 22] based techniques address some of the above issues, however, at the cost of accurate surface information [23, 9, 3, 13]. Recently, several end-to-end deep learning solutions for estimating the 3D parametric body model from a monocular image have been proposed  [11, 26, 27, 19, 18, 32]. They all attempt to estimate the pose (relative axis-angles) and shape parameters of the SMPL [14] body model, which is a complex non-linear mapping. To get around this complex mapping, several methods transform them to rotation matrices  [18, 19] or learn from the 2D/3D keypoint and silhouettes projections (a function of the parameters) [18, 11, 19]. Additionally,  [11] proposes an alternate method for training (Iterative Error Feedback) as well as a body joint specific adversarial losses, which takes upto 5 days to train. In other words, learning the parametric body model hasn’t been straightforward and has resulted in complex and indirect solutions that actually rely on different projections of the underlying mesh.

Directly regressing to point cloud or mesh data from image(s) is a severely ill-posed problem and there are very few attempts in deep learning literature in this direction [31, 17]. With regard to point cloud regression, most of the attempts have focused on rigid objects, where learning is done in a class specific manner. Apart from a very recent work [12], learning a mesh hasn’t been explored much for reconstruction primarily because of lack of deep learning constructs to do so.

In this paper, we attempt to work in between a generic point cloud and a mesh - i.e., we learn an ”implicitly structured” point cloud. We hypothesize that in order to perform parametric body model based reconstruction, instead of learning the highly non-linear SMPL parameters, learning its corresponding point cloud (although high dimensional) and enforcing the same parametric template topology on it is an easier task. This is because, in SMPL like body models, each of the surface vertices are a sparse linear combination of the transformations induced by the underlying joints i.e., implicitly learning the skinning function by which parametric models are constructed is easier than learning the non-linear axis-angle representation itself (parameters). Further, such models lack high-resolution local surface details as well. Therefore, there are far fewer ”representative” points that we have to learn. In comparison with generic point cloud regression as well, this is an easier task because of this implicit structure that exists between these points.

Going ahead, attempting to produce high resolution meshes are a natural extension that is easier in 3D space than in the parametric one. Therefore, we believe that this is a direction worth exploring and we present an initial solution in that direction - HumanMeshNet that simultaneously performs shape estimation by regressing to template mesh vertices (by minimizing surface loss) as well receives a body pose regularisation from a parallel branch in multi-task setup. The image to mesh vertex regression is further explicitly conditioned on the neighborhood constraint imposed by the mesh topology, thus ensuring a smooth surface reconstruction. Figure 2 outlines the architecture of HumanMeshNet.

Ours is a relatively simpler model as compared to majority of the existing methods for volumetric and parametric model prediction (e.g., [28]). This makes it efficient in terms of network size as well as feed forward time yielding significantly high frame-rate reconstructions. At the same time, our simpler network achieves comparable accuracy in terms of surface and joint error w.r.t. majority of state-of-the-art techniques on three publicly available datasets. The proposed paradigm can theoretically learn local surface deformations induced by body shape variations which the PCA space of parametric body models can’t capture. In addition to predicting the body model, we also show the generalizability of our proposed idea for solving a similar task with different structure - non-rigid hand mesh reconstructions from a monocular image.

To summarize, the key contributions of this work are:

• •

We propose a simple end-to-end multi-branch, multi-task deep network that exploits a ”structured point cloud” to recover a smooth and fixed topology mesh model from a monocular image.

• •

The proposed paradigm can theoretically learn local surface deformations induced by body shape variations which the PCA space of parametric body models can’t capture.

• •

The simplicity of the model makes it efficient in terms of network size as well as feed forward time yielding significantly high frame-rate reconstructions, while simultaneously achieving comparable accuracy in terms of surface and joint error, as shown on three publicly available datasets.

• •

We also show the generalizability of our proposed paradigm for a similar task of reconstructing the hand mesh models from a monocular image.

2 Related Work

Estimating 3D Body Models: The traditional approach for parametric body model fitting entails iteratively optimizing an objective function with 2D supervision in the form of silhouettes, 2D key points etc [23, 9, 3, 13]. However, they often involve manual intervention and are time-consuming to solve as well as susceptible to converge at local optima.

On the deep learning front,  [11] proposes an iterative regression with 3D and 2D joint loss as a feedback and an adversarial supervision for each joint. However, this architecture has a large number of networks and hence is not easy to train.  [18] predicts a colour-coded body segmentation that is used as a prior to predict the parameters. Similarly, in [19], 2D heatmaps and silhouettes are predicted first, which are then used to predict the pose and shape parameters. All of the above methodologies calculate loss on 2D keypoints or silhouette projections of the rendered mesh, which significantly slows down training time (due to model complexity), in addition to requiring additional supervision.  [28] proposes a complex multi-task network with total of six networks (having respective losses computed on 2D and 3D joint locations, 2D segmentation mask, volumetric grid and sillhoute reprojection of volumetric and SMPL model). This makes it a significantly heavy network with a longer feed forward time. The focus of reconstruction is to retrieve the boundary of the subject in 3D space. However, in a volumetric representation, predicting the volume within the surface is counterproductive. On the other hand, we focus on direct image to mesh vertex regression for recovering the surface points. The most recent state-of-the-art work proposed in [12] also recovers sparse surface points using Graph Neural Network(GCN). However GCNs experience troubles learning the global structure because of its neighbourhood aggregation scheme [33].

Estimating Hand Models: While most of the hand recovery methods typically estimate the 3D pose from one or multiple RGB/Depth images, hand shape estimation hasn’t been extensively explored. For a detailed survey of the field, we refer to [25, 34]. Recent effort in [16] was the first attempt to predict both the pose and the vertex based full 3D mesh representation (surface shape) from a single depth image. The recently proposed MANO [22] model is an SMPL like model that describes both the shape and pose, and is learned from thousands of high resolution scans.  [5] predicted the MANO parameters from a monocular RGB image, but, they don’t show much shape variations.  [8] use a graph CNN to recover the hand surface from monocular RGB image of the hand.

3 Proposed Method: HumanMeshNet

In order to learn this structured point cloud, we use an encoder- and multi-decoder model, which we describe in this section. Figure 2 gives an overview of our end-to-end pipeline. Our model consists of three primary phases:

**Phase 1 - RGB to Partwise Segmentation: **Given an input RGB image of size 224x224, we first predict a discrete body part label for each pixel in the input image (for a total of 24 body parts) using just the body part labelling network from [1]. A part-wise segmentation enables a tracking of the human body in the image, making it easier for shape estimation.

**Phase 2 - Image Encoders and Joint Embedding: ** Both the RGB image and segmentation mask are passed through separate encoders, each a Resnet-18, and their respective CNN feature vectors, each of dimension 1000 are concatenated together to obtain a joint embedding.

Such fusion of RGB and segmentation mask was employed to combine complementary information from each modality. This is important as segmentation mask predictions can be very noisy in many scenarios (see Figure 3), e.g., low lighting, distance of the person from the camera, sensing noise etc., leading to failures like interchanged limbs or missing limbs.

Phase 3 - Multi-branch Predictions: From our concatenated feature embedding, we branch out into two complimentary tasks via Fully Connected layers (FCs). Each branch consists of two FCs, each of dimension 1000 followed by the respective output dimensions for the 3D joints, and 3D surface respectively. It is to be noted that our predictions are in camera frame.

Loss Function. We use a multi-branch loss functions to train our network i.e, $L_{S}$, $L_{J}$ and $L_{JS}$. We regularized the loss functions such that they contribute equally to the overall loss. This translates to Equation 1.

[TABLE]

The surface loss $L_{S}$ in Equation 2 gives the vertex-wise Euclidean distance between the predicted vertices $V_{i}$ and ground truth vertices $\hat{V_{i}}$ for the 3D mesh prediction branch in Figure 2 (e).

[TABLE]

However, this loss does not ensure prediction of smooth surfaces as each vertex is independently predicted.

Nevertheless, each mesh vertex has a neighborhood structure that can be used to further refine the estimate of individual vertex. Here we make use of smoothing regularisation [24] (as shown in Equation 3), where the position of each vertex, $V_{i}$, is replaced by the average position, of its neighbours $N(V_{j})$.

[TABLE]

This is achieved by first applying the smoothness mesh regularization given by Equation 3 and then calculating $L_{S}$. This helps in limiting the number of surface jitters or irregularities.

In order to enforce 3D joints consistency, we minimize joint loss $L_{J}$ defined in Equation 4, which gives the euclidean distance between the predicted joints $J_{i}$ and ground truth joints $\hat{J_{i}}$ in the 3D joint prediction branch as shown in Figure 2(d).

[TABLE]

The 3D joints $JS_{i}$ under the surface are recovered using the SMPL joint regressor [14]. We also minimize the loss $L_{JS}$ defined in Equation 5 which gives the euclidean distance between the joints $J_{i}$ predicted from the joints branch and the joints $JS_{i}$ from the surface branch. It helps both the branches to learn consistently with each other.

[TABLE]

Network Variants: We define two different variants of HumanMeshNet in order to perform an extensive analysis:

1. (a)

HumanMeshNet (HMNet) - The base version which uses an ”off-the-shelf” body part segmentation network ( [1]). 2. (b)

HumanMeshNetOracle (HMNetOracle) - A refined version using a more accurate body part segmentation given by the dataset. However, in some datasets (e.g., UP-3D, [13]), these segmentation masks can be noisy due to manual annotations.

4 Experiments & Results

In this section, we show a comprehensive evaluation of the proposed model and benchmark against the state-of-the-art optimization and deep learning based Parametric (P), Volumetric (V) and Surface based (S) reconstruction algorithms. It is to be noted that we train on each dataset separately and report on its given test sets. All of the trained models and code shall be made publicly available, along with a working demo. Please view our supplementary video for more results.

4.1 Datasets

SURREAL [29]: This dataset provides synthetic image examples with 3D shape ground truth. The dataset draws poses from MoCap [10] and body shapes from body scans [21] to generate valid SMPL instances for each image. Although this dataset is synthetically generated, it emulates complex real poses and shapes, coupled with challenging input images that contain background clutter and are reflective with low resolution. It has total 1.6 million training and 15,000 test samples.

UP-3D [13]: It is a recent dataset that collects color images from 2D human pose benchmarks and uses an extended version of SMPLify [3] to provide 3D human shape candidates. The candidates were evaluated by human annotators to select only the images with good 3D shape fits. It comprises of 8,515 images, where 7,126 are used for training and 1,389 for testing. However, the ground truth meshes are sometimes inaccurately generated as shown in Figure 4. We separately train the network and report results on full test set of UP-3D.

Human3.6M [10]: It is a large-scale pose dataset that contains multiple subjects performing typical actions like ”eating” and ”walking” in a lab environment. It consists of a downsampled version of the original data with 300,000 image-3D joint pairs for training and 100,000 such for testing. Since ground truth 3D meshes for any of the commonly reported protocols [4] for evaluation aren’t available anymore, we finetune SURREAL-pretrained network using joint loss only. We report the joint reconstruction error (trained as per Protocol 2 of  [4]) and therefore compare with those methods that don’t use mesh supervision for this dataset in Table 3.

4.2 Implementation Details

Data Pre-processing We use the ground truth bounding boxes from each of the datasets to obtain a square crop of the human. This is a standard step performed by most comparative 3D human reconstruction models.

**Network Training **We use Nvidia’s GTX 1080Ti, with 11GB of VRAM to train our models. A batch size of 64 is used for SURREAL and Human 3.6M datasets, and a batch size of 16 for UP3D dataset. We use the ADAM optimizer having an initial learning rate of $10^{-4}$, to get optimal performance. Attaining convergence on the SURREAL and Human3.6M takes 18 hours each, while on UP-3D takes 6 hours. We use the standard splits given by the datasets, for benchmarking, as indicated in Section 4.1.

Procrustes Analysis (PA) In order to evaluate the quality of the reconstructed mesh, we also report results after solving the Orthogonal Procrustes problem [21], in which we scale the output to the size of the ground truth and solve for rotation. Additionally, we also quantitatively evaluate without this alignment.

Evaluation Metric

1. (a)

Surface Error (mm): Gives the mean-per-vertex error between the ground truth and predicted mesh. 2. (b)

Joint Error (mm): Gives the mean-per joint error between the ground truth and predicted joints. All reported results are obtained from the underlying joints of the mesh, rather than the alternate branch, unless otherwise mentioned. 3. (c)

PA. Surface/Joint Error (mm): It is the surface/joint error after Procrustes Analysis (PA).

4.3 Comparison with State-of-the-art

**Baseline **We define our baseline as the direct prediction of a point cloud from an RGB image, using a Resnet-50. This enables us to show the novelty introduced by our pipeline and the usefulness of learning in this output space.

**Results & Discussion ** For qualitative results on all of the three datasets, refer to Figures 5,  6. A large amount of training data is required to learn a vast range of poses and shapes. However,  [29, 19] show a good domain transfer to real data by training on the synthetic SURREAL dataset. Since our supervision is dominated by surface meshes, SURREAL plays an important role in benchmarking our method. We show comparable performance on it, as indicated by Table 2. In Table 2, we also show our results with a subsampled mesh (subsampled as per [12]) from 6890 to 1723 vertices with almost no change in reconstruction error. This is a good proof of our hypothesis that there are far fewer representative points to learn in this structured point cloud.

UP-3D is an ”in the wild” dataset, however has inaccurate ground truth mesh annotations, as shown in Figure 4. Most circumvent this issue, by avoiding 3D supervision altogether and projecting back to a silhouette or keypoints [11, 19]. Further, training on such a small dataset doesn’t provide a good generalisation. Therefore, we observe a higher error in HMNet. However, HMNetOracle produces a significant increase in accuracy with the increase in quality of the input image and segmentation mask (Table 4). Similar to state of the art methods  [28, 30, 12], we rely on 3D body supervision and providing more supervision like sillhoute and 2D keypoint loss like  [28, 11] can improve the performance further. For Human3.6m, we compare with those that don’t use mesh supervision (since this data is currently unavailable) and achieve comparable performance.

4.4 Discussion

**Ablation Study: ** Directly regressing the mesh from RGB leads to sub-par performance. Limbs are typically the origin of maximum error in reconstruction, and the segmentation mask provides a better tracking in scenarios such as leg-swap shown in Figure 7. The first two rows of Table 4 quantitatively explain this behaviour. Further, by having a more accurate segmentation mask, HMNetOracle achieves a significant reduction in surface error ($\downarrow 34.7mm$). In scenarios with inaccurate ground truth 3D (Figure 4), the regularisation 3D joint loss in our multi-branch setup helps us in recovering better fits (row 4 for UP3D). In datasets such as Human3.6m where accurate MoCap markers are given, this multi-branch loss provides a good boost - with and without joint loss, the joint reconstruction error is 60.9mm v/s 67.3mm respectively.

Effect of Mesh Regularisation Our mesh regularization module adds a smoothing effect while training, therefore ensuring that the entire local patch should move towards the ground truth for minimizing the error. Although intrinsic geometry based losses can also be used here, we hypothesize that they have a larger impact when more complex local surface deformations (e.g., facial expressions) are present. Figure 8 show the impact of this regularization.

**Recovering Shape Variations ** Most parametric models prediction work with a neutral template model [11], and would have to learn the gender from the image. In our method, a direct mesh regression can learn the local shape variations (as long as training data has such variations) which extends to inherently learning gender invariant meshes. Two such samples are showing in Figure 9.

Generalizability to Hand Mesh Models. We show the generalizability of our model to a similar task with a different strcuture. First, we populated a SURREAL like synthetic hand dataset using the MANO hand model  [22], similar to  [8] with a total of 70,000 image-mesh pairs. We train our model on this dataset to predict hand surface and joints from an input RGB image using the same pipeline described in Figure 2. The training setting remains the same as earlier, and we obtain impressive qualitative results as shown in Figure 10. The average surface error across the test dataset is 1mm, which acts as a proof of concept that polygonal mesh reconstruction of non-rigid hands (although in a simplistic scenario), is feasible.

Network Runtime. Table 5 list out run-time of various methods. Comparing this with HMNet with HMNetOracle, it is evident that a major part of HMNet’s complexity arises from the multi-human pixel wise class prediction, which runs at around 30 FPS for an image of size 224x224. [6] is an accurate real time body part segmentation network which runs at 120 FPS, and can be incorporated into our system to produce accurate, real time reconstructions.

Limitations and Future Work. Since we do not enforce any volume consistency, skewing/thinning artifacts might be introduced in our meshes. We would like to account for these in a non-handcrafted anthropomorphically valid way by either learning the SMPL parameters on top of it using an MLP similar to [12] or by using a GAN to penalize fake/invalid human meshes. Further, we have made use of the mesh topology in two ways in this work - (a) implicitly, to make the learning easier and (b) for smoothing. Going ahead, we would like to make use of the mesh topology and geometry details is a more explicit manner, by using intrinsic mesh/surface properties. We believe that this is a largely unexplored space and applying such a regularization can result better exploitation of surface geometry for reconstruction.

5 Conclusion

We proposed a multi-branch multi-task HumanMeshNet network that simultaneously regress to the template mesh vertices as well as body joint locations from a single monocular image. The proposed method achieves comparable performance with significantly lower modelling and computational complexity on three publicly available datasets. We also show the generalizability of the proposed architecture for similar task of predicting the mesh of the hand. Looking forward, we would like to exploit intrisinc mesh properties to recover a more accurate surface reconstruction.