3DPeople: Modeling the Geometry of Dressed Humans

TL;DR

This paper introduces 3DPeople, a large synthetic dataset and a novel shape parameterization method, enabling end-to-end deep learning for modeling dressed human geometry from single images.

Contribution

It presents a new dataset, a spherical area-preserving parameterization algorithm, and an end-to-end deep network for predicting clothed human shapes from images.

Findings

High-quality 3D shape predictions from single images

Improved shape representation with spherical area-preserving maps

Effective modeling of clothing and pose in diverse scenarios

Abstract

Recent advances in 3D human shape estimation build upon parametric representations that model very well the shape of the naked body, but are not appropriate to represent the clothing geometry. In this paper, we present an approach to model dressed humans and predict their geometry from single images. We contribute in three fundamental aspects of the problem, namely, a new dataset, a novel shape parameterization algorithm and an end-to-end deep generative network for predicting shape. First, we present 3DPeople, a large-scale synthetic dataset with 2.5 Million photo-realistic images of 80 subjects performing 70 activities and wearing diverse outfits. Besides providing textured 3D meshes for clothes and body, we annotate the dataset with segmentation masks, skeletons, depth, normal maps and optical flow. All this together makes 3DPeople suitable for a plethora of tasks. We then represent the 3D shapes using 2D geometry images. To build these images we propose a novel spherical area-preserving parameterization algorithm based on the optimal mass transportation method. We show this approach to improve existing spherical maps which tend to shrink the elongated parts of the full body models such as the arms and legs, making the geometry images incomplete. Finally, we design a multi-resolution deep generative network that, given an input image of a dressed human, predicts his/her geometry image (and thus the clothed body shape) in an end-to-end manner. We obtain very promising results in jointly capturing body pose and clothing shape, both for synthetic validation and on the wild images.

Equations

$${\mathcal{G}}^Q={\mathcal{G}}^R\circ \mathcal{N}\circ \mathcal{I}:\mathbf{Q}\to \mathbf{X}$$

$${\mathcal{L}}_{\text{R}}={\mathbb{E}}_{\mathbf{X}\sim {\mathbb{P}}_r,\hat{\mathbf{X}}\sim {\mathbb{P}}_g}\frac{1}{S}\sum _{s=1}^S{\lambda }_s{\Vert {\mathbf{X}}_s-{\hat{\mathbf{X}}}_s\Vert }_1,$$

$${\mathcal{L}}_{\text{P}}={\mathbb{E}}_{\mathbf{X}\sim {\mathbb{P}}_r,\hat{\mathbf{X}}\sim {\mathbb{P}}_g}\frac{1}{S}\sum _{s=1}^S{\lambda }_s{\Vert \mathcal{P}({\mathbf{X}}_s)-\mathcal{P}({\hat{\mathbf{X}}}_s)\Vert }_1$$

$$\displaystyle\sum_{k=1}^{K}\Bigl{[}$$

$$\displaystyle+\frac{\lambda_{\text{dgp}}}{2}\mathbb{E}_{\mathbf{X}\sim\mathbb{P}_{r}}(\|\nabla D^{k}(\mathbf{X}_{S})\|_{1}^{2}\Bigr{]},$$

$${\mathbb{E}}_{\mathbf{X}\sim {\mathbb{P}}_r,\hat{\mathbf{X}}\sim {\mathbb{P}}_g}\sum _{k=1}^K\sum _{l=1}^L\frac{1}{N_l^k}{\Vert D_l^k({\mathbf{X}}_S)-D_l^k({\hat{\mathbf{X}}}_S)\Vert }_1,$$

$${\Phi }^{\star }=\arg \underset{\Phi }{\min }\underset{D}{\max }{\mathcal{L}}_{\text{adv}}+{\lambda }_{\text{R}}{\mathcal{L}}_{\text{R}}+{\lambda }_{\text{P}}{\mathcal{L}}_{\text{P}}+{\lambda }_{\text{F}}{\mathcal{L}}_{\text{F}}$$

$$\text{dist}(\hat{\mathbf{Y}},\mathbf{Y})=\frac{1}{2}(\text{KNN}(\hat{\mathbf{Y}}\to \mathbf{Y})+\text{KNN}(\mathbf{Y}\to \hat{\mathbf{Y}}))$$

Full text

First, we present 3DPeople, a large-scale synthetic dataset with 2.5 Million photo-realistic images of 80 subjects performing 70 activities and wearing diverse outfits. Besides providing textured 3D meshes for clothes and body, we annotate the dataset with segmentation masks, skeletons, depth, normal maps and optical flow. All this together makes 3DPeople suitable for a plethora of tasks.

We then represent the 3D shapes using 2D geometry images. To build these images we propose a novel spherical area-preserving parameterization algorithm based on the optimal mass transportation method. We show this approach to improve existing spherical maps which tend to shrink the elongated parts of the full body models such as the arms and legs, making the geometry images incomplete.

Finally, we design a multi-resolution deep generative network that, given an input image of a dressed human, predicts his/her geometry image (and thus the clothed body shape) in an end-to-end manner. We obtain very promising results in jointly capturing body pose and clothing shape, both for synthetic validation and on the wild images.

1 Introduction

With the advent of deep learning, the problem of predicting the geometry of the human body from single images has experienced a tremendous boost. The combination of Convolutional Neural Networks with large MoCap datasets [42, 19], resulted in a substantial number of works that robustly predict the 3D position of the body joints [27, 28, 30, 34, 38, 46, 49, 52, 59].

In order to estimate the full body shape a standard practice adopted in [11, 13, 17, 22, 50, 61] is to regress the parameters of low rank parametric models [9, 26]. Nevertheless, while these parametric models describe very accurately the geometry of the naked body, they are not appropriate to capture the shape of clothed humans.

Current trends focus on proposing alternative representations to the low rank models. Varol et al. [51] advocate for a direct inference of volumetric body shape, although still without accounting for the clothing geometry. Very recently, [33] uses 2D silhouettes and the visual hull algorithm to recover shape and texture of clothed human bodies. Despite very promising results, this approach still requires frontal-view input images of the person with no background, and under relatively simple body poses.

In this paper, we introduce a general pipeline to estimate the geometry of dressed humans which is able cope with a wide spectrum of clothing outfits and textures, complex body poses and shapes, and changing backgrounds and camera viewpoints. For this purpose, we contribute in three key areas of the problem, namely, the data collection, the shape representation and the image-to-shape inference.

Concretely, we first present 3DPeople a new large-scale dataset with 2.5 Million photorealistic synthetic images of people under varying clothes and apparel. We split the dataset 40 male/40 female with different body shapes and skin tones, performing 70 distinct actions (see Fig. 1). The dataset contains the 3D geometry of both the naked and dressed body, and additional annotations including skeletons, depth and normal maps, optical flow and semantic segmentation masks. This additional data is indeed very similar to SURREAL [52] which was built for similar purposes. The key difference between SURREAL and 3DPeople, is that in SURREAL the clothing is directly mapped as a texture on top of the naked body, while in 3DPeople the clothing does have its own geometry.

As essential as gathering a rich dataset, is the question of what is the most appropriate geometry representation for a deep network. In this paper we consider the “geometry image” proposed originally in [16] and recently used to encode rigid objects in [7, 43]. The construction of the geometry image involves two steps, first a mapping of a genus-0 surface onto a spherical domain, and then to a 2D grid resembling an image. Our contribution here is on the spherical mapping. We found that existing algorithms [12, 7] were not accurate, specially for the elongated parts of the body. To address this issue we devise a novel spherical area-preserving parameterization algorithm that combines and extends the FLASH [12] and the optimal mass transportation methods [31].

Our final contribution consists of designing a generative network to map input RGB images of a dressed human into his/her corresponding geometry image. Since we consider $128\times 128\times 3$ geometry images, learning such a mapping is highly complex. We alleviate the learning process through a coarse-to-fine strategy, combined with a series of geometry-aware losses. The full network is trained in an end-to-end manner, and the results are very promising in variety of input data, including both synthetic and real images.

2 Related work

3D Human shape estimation. While the problem of localizing the 3D position of the joints from a single image has been extensively studied  [27, 28, 30, 34, 38, 46, 49, 52, 59, 62], the estimation of the 3D body shape has received relatively little attention. This is presumably due to the existence of well-established datasets [42, 19], uniquely annotated with skeleton joints.

The inherent ambiguity for estimating human shape from a single view is typically addressed using shape embeddings learned from body scan repositories like SCAPE [9] and SMPL [26]. The body geometry is described by a reduced number of pose and shape parameters, which are optimized to match image characteristics [10, 11, 25]. Dibra et al. [13] are the first in using a CNN fed with silhouette images to estimate shape parameters. In [47, 50] SMPL body parameters are predicted by incorporating differential renders into the deep network to directly estimate and minimize the error of image features. On top of this,  [22] introduces an adversarial loss that penalizes non-realistic body shapes. All these approaches, however, build upon low-dimensional parametric models which are only suitable to model the geometry of the naked body.

Non-parametric representations for 3D objects. What is the most appropriate 3D object representation to train a deep network remains an open question, especially for non-rigid bodies. Standard non-parametric representations for rigid objects include voxels [14, 58], octrees [48, 54, 55] and point-clouds [45].  [7, 43] uses 2D embeddings computed with geometry images [16] to represent rigid objects. Interestingly, very promising results for the reconstruction of non-rigid hands were also reported. DeformNet [36] proposes the first deep model to reconstruct the 3D shape non-rigid surfaces from a single image. Bodynet [51] explores a network that predicts voxelized human body shape. Very recently, [33] introduces a pipeline that given a single image of a person in frontal position predicts the body silhouette as seen from different views, and then uses a visual hull algorithm to estimate 3D shape.

Generative Adversarial Networks. Originally introduced by [15], GANs have been previously used to model human body distributions and generate novel images of a person under arbitrary poses [37]. Kanazawa et al. [22] explicitly learned the distribution on real parameters of SMPL. DVP [23], paGAN [32] and GANimation [35] presented models for continuous face animation and manipulation. They have also been applied to edit [18, 44, 53] and generate [5] talking faces.

Datasets for body shape analysis. Datasets are fundamental in the deep-learning era. While obtaining annotations is quite straightforward for 2D poses [41, 8, 21], it requires using sophisticated MoCap systems for the 3D case. Additionally, the datasets acquired this way [42, 19, 19] are mostly indoors. Even more complex is the task of obtaining 3D body shape, which requires expensive setups with muti-cameras or 3D scanners. To overcome this situation, datasets with synthetically but photo-realistic images have emerged as a tool to generate massive amounts of training data. SURREAL [52] is the largest and more complete dataset so far, with more than 6 million frames generated by projecting synthetic textures of clothes onto random SMPL body shapes. The dataset is further annotated with body masks, optical flow and depth. However, since clothes are projected onto the naked SMPL shapes just as textures, they can not be explicitly modeled. To fill this gap, we present the 3DPeople dataset of 3D dressed humans in motion.

3 3DPeople dataset

To facilitate future researches, we introduce 3DPeople, the first dataset of dressed humans with specific geometry representation for the clothes. The dataset in numbers can be summarized as follows: it contains 2.5 Million photorealistic $640\times 480$ images split into 40 male/40 female performing 70 actions. Every subject-action sequence is captured from 4 camera views, and for each view the texture of the clothes, the lighting direction and the background image are randomly changed. Each frame is annotated with (see Fig. 1): 3D textured mesh of the naked and dressed body; 3D skeleton; body part and cloth segmentation masks; depth map; optical flow; and camera parameters. We next briefly describe the generation process:

Body models: We have generated fully textured triangular meshes for 80 human characters using Adobe Fuse [1] and MakeHuman [2]. The distribution of the subjects physical characteristics cover a broad spectrum of body shapes, skin tones and hair geometry (see Fig. 1).

Clothing models: Each subject is dressed with a different outfit including a variety of garments, combining tight and loose clothes. Additional apparel like sunglasses, hats and caps are also included. The final rigged meshes of the body and clothes contain approximately 20K vertices.

Mocap squences: We gather 70 realistic motion sequences from Mixamo [3]. These include human movements with different complexity, from drinking and typing actions that produce small body motions to actions like breakdance or backflip that involve very complex patterns. The mean length of the sequences is of 110 frames. While these are relatively short sequences, they have a large expressivity, which we believe make 3DPeople also appropriate for exploring action recognition tasks.

Textures, camera, lights and background: We then use Blender [4] to apply the 70 MoCap animation sequences to each character. Every sequence is rendered from 4 camera views, yielding a total of 22,400 clips. We use a projective camera with a 800 mm focal length and $640\times 480$ pixel resolution. The 4 viewpoints correspond approximately to orthogonal directions aligned with the ground. The distance to the subject changes for every sequence to ensure a full view of the body in all frames. The textures of the clothes are randomly changed for every sequence (see again Fig. 1). The illumination is composed of an ambient lighting plus a light source at infinite, which direction is changed per sequence. As in [52] we render the person on top of a static background image, randomly taken from the LSUN dataset [60].

Semantic labels: For every rendered image, we provide segmentation labels of the clothes (8 classes) and body (14 parts). Observe in Fig. 1-top-right that the former are aligned with the dressed human, while the body parts are aligned with the naked body.

4 Problem formulation

Given a single image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ of a person wearing an arbitrary outfit, we aim at designing a model capable of directly estimating the 3D shape of the clothed body. We represent the body shape through the mesh associated to a geometry image with $N^{2}$ vertices $\mathbf{X}\in\mathbb{R}^{N\times N\times 3}$ where $\mathbf{x}_{i}=(x_{i},y_{i},z_{i})$ are the 3D coordinates of the $i$-th vertex, expressed in the camera coordinates system and centered on the root joint $\mathbf{x}_{r}$. This representation is a key ingredient of our design, as it maps the 3D mesh to a regular 2D grid structure that preserves the neighborhood relations, fulfilling thus the locality assumption required in CNN architectures. Furthermore, the geometry image representation allows uniformly reducing/increasing the mesh resolution by simply uniformly downsampling/upsampling. This will play an important role in our strategy of designing a coarse-to-fine shape estimation approach.

We next describe the two main steps of our pipeline: 1) the process of constructing the geometry images, and 2) the deep generative model we propose for predicting 3D shape.

5 Geometry image for dressed humans

The deep network we describe later will be trained using pairs $\{\mathbf{I},\mathbf{X}\}$ of images and their corresponding geometry image. For creating the geometry images we consider two different cases, one for a reference mesh in a tpose configuration, and another for any other mesh of the dataset.

5.1 Geometry image for a reference mesh

One of the subjects of our dataset in a tpose configuration is chosen as a reference mesh. The process for mapping this mesh into a planar regular grid is illustrated in Fig. 2. It involves the following steps:

Repairing the mesh. Let $\mathbf{R}^{\textrm{tpose}}\in\mathbb{R}^{N_{R}\times 3}$ be the reference mesh with $N_{R}$ vertices in a tpose configuration (Fig. 2-a). We assume this mesh to be a manifold mesh and to be genus-0. Most of the meshes in our dataset, however, do not fulfill these conditions. In order to fix the mesh we follow the heuristic described in [7] which consists of a voxelization, a selection of the largest connected region of the $\alpha$-shape, and subsequent hole filling using a medial axis approach. We denote by $\tilde{\mathbf{R}}^{\textrm{tpose}}$ the repaired mesh.

Spherical parameterization. Given the repaired genus-0 mesh $\tilde{\mathbf{R}}^{\textrm{tpose}}$, we next compute the spherical parameterization $\mathcal{S}:\tilde{\mathbf{R}}^{\textrm{tpose}}\rightarrow\mathbf{S}$ that maps every vertex of $\tilde{\mathbf{R}}^{\textrm{tpose}}$ onto the unit sphere $\mathbf{S}$ (Fig. 2-b). Details of the algorithm we use are explained below.

Unfolding the sphere. The sphere $\mathbf{S}$ is mapped onto an octahedron and then cut along edges to output a flat geometry image $\mathbf{X}$. Let us formally denote by $\mathcal{U}:\mathbf{S}\rightarrow\mathbf{X}$, and by $\mathcal{G}^{R}=\mathcal{U}\circ\mathcal{S}:\tilde{\mathbf{R}}^{\textrm{tpose}}\rightarrow\mathbf{X}$ the mapping from the reference mesh to the geometry image. The unfolding process is shown in Fig. 2-(c,d,e). Color lines in the geometry image correspond to the same edge in the octahedron, and are split after the unfolding operation. We will later enforce this symmetry constraint when predicting geometry images.

5.2 Spherical area-Preserving parameterization

Although there exist several spherical parameterization schemes (e.g. [12, 7]) we found that they tend to shrink the elongated parts of the full body models such as the arms and legs, making the geometry images incomplete (see Fig. 3). In this work, we develop a spherical area-preserving parameterization algorithm for genus-0 full body models by combining and extending the FLASH method [12] and the optimal mass transportation method [31]. Our algorithm is particularly advantageous for handling models with elongated parts. The key idea is to begin with an initial parameterization onto a planar triangular domain with a suitable rescaling correcting the size of it. The area distortion of the initial parameterization is then reduced using quasi-conformal composition. Finally, the spherical area-preserving parameterization is produced using optimal mass transportation followed by the inverse stereographic projection.

5.3 Geometry image for arbitrary meshes

The approach for creating the geometry image described in the previous subsection is quite computationally demanding (can last up to 15 minutes for meshes with complex topology). To compute the geometry image for several thousand training meshes we have devised an alternative approach. Let $\mathbf{Q}\in\mathbb{R}^{N_{Q}\times 3}$ be the mesh of any subject of the dataset under an arbitrary pose (Fig. 4-a), and let $\mathbf{Q}^{\textrm{tpose}}\in\mathbb{R}^{N_{Q}\times 3}$ be its tpose configuration (Fig. 4-b). We assume there is a 1-to-1 vertex correspondence between both meshes, that is, $\exists\hskip 5.69054pt\mathcal{I}:\mathbf{Q}\rightarrow\mathbf{Q}^{\textrm{tpose}}$ where $\mathcal{I}$ is a known bijective function111This is guaranteed in our dataset, with all meshes of the same subject having the same number of vertices..

We then compute dense correspondences between $\mathbf{Q}^{\textrm{tpose}}$ and the reference tpose $\tilde{\mathbf{R}}^{\textrm{tpose}}$, using a nonrigid icp algorithm [6]. We denote this mapping as $\mathcal{N}:\mathbf{Q}^{\textrm{tpose}}\rightarrow\tilde{\mathbf{R}}^{\textrm{tpose}}$ (see Fig. 4-c). We can then finally compute the geometry image for the input mesh $\mathbf{Q}$ by concatenating mappings:

[TABLE]

where $\mathcal{G}^{R}$ is the mapping from the reference mesh to the geometry image domain estimated in Sec. 5.1. It is worth pointing the the nonrigid icp between the pairs of tposes is also highly computationally demanding, but it only needs to be computed once per every subject of the dataset. Once this is done, the geometry image for a new input mesh $\mathbf{Q}$ can be created in a few seconds.

An important consequence of this procedure is that all geometry images of the dataset will be semantically aligned, that is, every $uv$ entry in $\mathbf{X}$ will correspond to (approximately) the same semantic part of the model. This will significantly alleviate the learning task of the deep network.

6 GimNet

We next introduce GimNet, our deep generative network to estimate geometry images (and thus 3D shape) of dressed humans from a single images. An overview of the model is shown in Fig. 5. Given the input image, we first extract the 2D joint locations $\mathbf{p}$ represented as heatmaps [57, 36], which are then fed into a mesh regressor $\Phi(\mathbf{I},\mathbf{p})$ trained to reconstruct the shape $\hat{\mathbf{X}}$ of the person in $\mathbf{I}$ employing a geometry image based representation. Due to the high complexity of the mapping (both $\mathbf{I}$ and $\hat{\mathbf{X}}$ are of size $128\times 128\times 3$), the regressor operates in a coarse-to-fine manner, progressively reconstructing meshes at higher resolution. To further enforce the reconstruction to lie on the manifold of anthropomorphic shapes, an adversarial scheme with two discriminators $D$ is applied.

6.1 Model architecture

Mesh regressor. Given the input image $\mathbf{I}$ and the estimated 2D body joints $\mathbf{p}$, the mesh regressor $\Phi$ aims to predict the geometry image $\mathbf{X}$, i.e. we seek to estimate the mapping $\mathcal{M}:\mathbf{I},\mathbf{p}\rightarrow\mathbf{X}$. Instead of directly learning the complex mapping $\mathcal{M}$, we break the process into a sequence of more manageable steps. $\Phi$ initially estimates a low-resolution mesh, and then progressively increases its resolution (see Fig. 5). This coarse-to-fine approach allows the regressor to first focus on the basic shape configuration and then shift attention to finer details, while also providing more stability compared to a network that learns the direct mapping.

As shown in Fig. 2-e, the geometry images have symmetry properties derived from unfolding the octahedron into a square, specifically, each side of the geometry image is symmetric with respect to its midpoint. We force this property using a differentiable layer that linearly operates over the edges of the estimated geometry images.

Multi-Scale Discriminator. Evaluating high-resolution meshes poses a significant challenge for a discriminator, as it needs to simultaneously guarantee local and global mesh consistency on very high dimensional data. We therefore use two discriminators with the same architecture, but that operate in different geometry image scales: (i) a discriminator with a large receptive field that evaluates the shape coherence as a whole; and (ii) a local discriminator that focuses on small patches and enforces the local consistency of the surface triangle faces.

6.2 Learning the model

3D reconstruction error. We first define a supervised multi-level L1 loss for 3D reconstruction $\mathcal{L}_{\text{R}}$ as:

[TABLE]

being $\mathbb{P}_{r}$ and $\mathbb{P}_{g}$ the real and generated data distribution of clothed human geometry images respectively, $S$ the number of scales, $\mathbf{X}_{s}$ the ground-truth reconstruction at scale $s$ and $\hat{\mathbf{X}}_{s}=\Phi_{s}(\mathbf{I})$ the estimated reconstruction. The error at each scale is weighted by $\lambda_{s}=\frac{1}{r}$ where $r$ is the ratio between $\hat{\mathbf{X}}_{S}$ and $\hat{\mathbf{X}}_{s}$ sizes. During initial experimentation L1 loss reported better reconstructions than mean squared error.

2D Projection Error. To encourage the mesh to correctly project onto the input image we penalize, at every scale $s$, its projection error $\mathcal{L}_{\text{P}}$ computed as:

[TABLE]

where $\mathcal{P}$ is the differentiable projection equation and $\lambda_{s}$ is calculated as above.

Adversarial loss. In order to further enforce the mesh regressor $\Phi$ to generate anthropomorphic shapes we perform a min-max strategy game [15] between the regressor and two discriminators operating at different scales. It is well-known that non-overlapping support between the true data distribution and model distributions can cause severe training instabilities. As proven by [40, 29], this can be addressed by penalizing the discriminator when deviating from the Nash-equilibrium, ensuring that its gradients are non-zero orthogonal to the data manifold. Formally, being $D^{k}$ the $k$th discriminator, the $\mathcal{L}_{\text{adv}}$ loss is defined as:

[TABLE]

where $\lambda_{\text{dgp}}$ is a penalty regularization for discriminator gradients, only considered on the true data distribution.

Feature matching loss. To improve training stabilization we penalize higher level features on the discriminators [56]. Similar to a perception loss, the estimated geometry image is compared with the ground truth at multiple feature levels of the discriminators. Being $D_{l}^{k}$ the $l$th layer of the $k$th discriminator, $\mathcal{L}_{\text{F}}$ is defined as:

[TABLE]

where $N_{l}^{k}$ is a weight regularizer denoting the number of elements in the $l$th layer of the $k$th discriminator.

Total Loss. Finally, we to solve the min-max problem:

[TABLE]

where $\lambda_{\text{R}}$, $\lambda_{\text{P}}$ and $\lambda_{\text{F}}$ are the hyper-parameters that control the relative importance of every loss term.

6.3 Implementation details

For the mesh regressor $\Phi$ we build upon the U-Net architecture [39] consisting on an encoder-decoder structure with skip connections between features at the same resolution extended to estimate geometry images at multiple scales. Both discriminator networks operate at different mesh resolutions [56] but have the same PatchGan [20] architecture mapping from the geometry image $\mathbf{X}$ to a matrix $\mathbf{Y}\in\mathbb{R}^{H/8\times W/8}$, where $\mathbf{Y}[i,j]$ represents the probability of the patch $ij$ to be close to a real geometry image distribution. The global discriminator evaluates the final mesh resolution at scale $S$ and the local discriminator the down-sampled mesh at scale $S-1$.

The model is trained with 170,000 synthetic images of cropped clothed people resized to $128\times 128$ pixels and geometry images of $128\times 128\times 3$ (meshes with 16,384 vertices) during 60 epochs and $S=4$. As for the optimizer, we use Adam [24] with learning rate of $2e-4$, beta1 $0.5$, beta2 $0.999$ and batch size $110$. Every $40$ epochs we decay the learning rate by a factor of $0.5$. The weight coefficients for the loss terms are set to $\lambda_{\text{R}}=20$, $\lambda_{\text{P}}=0.1$, $\lambda_{\text{F}}=10$ and $\lambda_{\text{dgp}}=0.01$.

7 Experimental evaluation

We next present quantitative and qualitative results on synthetic images of our dataset and on images in the wild.

Synthetic Results We evaluate our approach on 25,000 test images randomly chosen for 8 subjects (4 male/ 4 female) of the test split. For each test sample we feed GimNet with the RGB image and the ground truth 2D pose, corrupted by Gaussian noise with 2 pixel std. For a given test sample, let $\hat{\mathbf{Y}}$ be the $N^{2}\times 3$ estimated mesh, resulting from a direct reshaping of its estimated geometry image $\hat{\mathbf{X}}$. Also, let $\mathbf{Y}$ be the ground truth mesh, which does not need to have neither the same number of vertices as $\tilde{\mathbf{Y}}$, nor necessarily the same topology. Since there is no direct 1-to-1 mapping between the vertices of the two meshes we propose using the following metric:

[TABLE]

where $\textrm{KNN}(\hat{\mathbf{Y}}\rightarrow\mathbf{Y})$ represents the average Euclidean distance for all vertices of $\hat{\mathbf{Y}}$ to their nearest neighbor in $\mathbf{Y}$. Note that $\textrm{KNN}(\cdot,\cdot)$ is not a true distance measure because it is not symmetric. This is why we compute it bidirectionally.

The quantitative results are summarized in Fig. 6. We report the average error (in mm) of GimNet for 30 actions (the 15 with the highest and lowest error). Note that the error of GimNet is bounded between 15 and 35mm. Recall, however, that we do not consider outlier 2D detections in our experiments, but just 2D noise. We also evaluate the error of the ground truth geometry image, as it is an approximation of the actual ground truth mesh. This error is below 5mm, indicating that the geometry image representation does indeed capture very accurately the true shape. Finally, we also provide the error of the recent parametric approach of [22], that fits SMPL parameters to the input images. Nevertheless, these results are just indicative, and cannot be directly compared with our approach, as we did not retrain [22]. We add them here just to demonstrate the challenge posed by the new 3DPeople dataset. Indeed, the distance error in [22] was computed after performing a rigid-icp of the estimated mesh with the ground truth mesh (there was no need of this for GimNet).

Qualitative Results We finally show in Fig. 7 qualitative results on synthetic images from 3DPeople and real fashion images downloaded from Internet. Remarkably, note how our approach is able to reconstruct long dresses (top row images), known to be a major challenge [33]. Note also that some of the reconstructed meshes have some spikes. This is one of the limitations of the non-parametric models, that the reconstructions tend to be less smooth than when using parametric fittings. However, non-parametric models have also the advantage that, if properly trained, can span a much larger configuration space.

8 Conclusions

In this paper we have made three contributions to the problem of reconstructing the shape of dressed humans: 1) we have presented the first large-scale dataset of 3D humans in action in which cloth geometry is explicitly modelled; 2) we have proposed a new algorithm to perform spherical parameterizations of elongated body parts, to later model rigged meshes of human bodies as geometry images; and 3) we have introduced an end-to-end network to estimate human body and clothing shape from single images, without relying on parametric models. While the results we have obtained are very promising, there are still several avenues to explore. For instance, extending the problem to video, exploring new regularization schemes on the geometry images, or combining segmentation and 3D reconstruction are all open problems that can benefit from the proposed 3DPeople dataset.

Acknowledgments

This work is supported in part by an Amazon Research Award, the Spanish Ministry of Science and Innovation under projects HuMoUR TIN2017-90086-R, ColRobTransp DPI2016-78957 and María de Maeztu Seal of Excellence MDM-2016-0656; and by the EU project AEROARMS ICT-2014-1-644271. We also thank NVidia for hardware donation under the GPU Grant Program.