DEMEA: Deep Mesh Autoencoders for Non-Rigidly Deforming Objects

TL;DR

DEMEA introduces a novel deep mesh autoencoder with an embedded deformation layer that explicitly models non-rigid deformations, improving mesh modeling and reconstruction for deformable objects.

Contribution

The paper presents DEMEA, a new mesh autoencoder with an embedded deformation layer that decouples deformation parameterization from mesh resolution, enhancing modeling of non-rigid deformations.

Findings

Higher quality results for highly deformable objects.

Effective non-rigid 3D reconstruction from depth and shading.

Successful deformation transfer across different meshes.

Abstract

Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling. We propose a general-purpose DEep MEsh Autoencoder (DEMEA) which adds a novel embedded deformation layer to a graph-convolutional mesh autoencoder. The embedded deformation layer (EDL) is a differentiable deformable geometric proxy which explicitly models point displacements of non-rigid deformations in a lower dimensional space and serves as a local rigidity regularizer. DEMEA decouples the parameterization of the deformation from the final mesh resolution since the deformation is defined over a lower dimensional embedded deformation graph. We perform a large-scale study on four different datasets of deformable objects. Reasoning about the local rigidity of meshes using EDL allows us to achieve higher-quality results for highly deformable objects, compared to directly regressing vertex positions. We demonstrate multiple applications of DEMEA, including non-rigid 3D reconstruction from depth and shading cues, non-rigid surface tracking, as well as the transfer of deformations over different meshes.

Tables

Table 1. Table 1 : Number of vertices on each level of the mesh hierarchy. Bold levels denote the embedded graph. Note that except for Cloth these values were computed automatically based on [ 31 ] .

	Mesh	1st	2nd	3rd	4th
DFaust [4]	6890	1723	431	108	27
CoMA [31]	5023	1256	314	79	20
SynHand5M [26]	1193	299	75	19	5
Cloth [3]	961	256	100	36	16

Table 2. Table 3 : Average per-vertex errors on the test sets of DFaust ( c ​ m 𝑐 𝑚 cm ), SynHand5M ( m ​ m 𝑚 𝑚 mm ), textureless cloth ( m ​ m 𝑚 𝑚 mm ) and CoMA ( m ​ m 𝑚 𝑚 mm ) for 8 and 32 latent dimensions.

	DFaust		SynHand5M		Cloth		CoMA
	8	32	8	32	8	32	8	32
w/ GL	8.92	2.75	9.02	2.95	11.26	6.45	1.38	0.99
w/ LP	7.71	2.22	8.00	2.52	11.46	7.96	1.25	0.79
Ours	6.69	2.23	8.12	2.51	11.28	6.40	1.23	0.81

Table 3. Table S.1 : Artifacts. The top rows use 32 latent dimensions, the last two rows are for 8.

Convolutional Ablation	DEMEA

Table 4. Table S.2 : Artifacts on SynHand5M. In contrast to CA, DEMEA yields a smooth index finger in the examples shown above.

Convolutional Ablation 8	DEMEA 8	Convolutional Ablation 32	DEMEA 32

Table 5. Table S.3 : Average per-vertex errors on the test sets of DFaust (in c ​ m 𝑐 𝑚 cm ), SynHand5M (in m ​ m 𝑚 𝑚 mm ), textureless cloth (in m ​ m 𝑚 𝑚 mm ) and CoMA (in m ​ m 𝑚 𝑚 mm ) for 8 and 32 latent dimensions, including standard deviations across the test sets.

	DFaust		SynHand5M		Cloth		CoMA
	8	32	8	32	8	32	8	32
CA	6.35 $\pm$ 2.40	2.07 $\pm$ 0.73	8.12 $\pm$ 1.77	2.60 $\pm$ 0.60	11.21 $\pm$ 4.58	6.50 $\pm$ 1.85	1.17 $\pm$ 0.47	0.72 $\pm$ 0.22
MCA	6.21 $\pm$ 2.48	2.13 $\pm$ 0.79	8.11 $\pm$ 1.77	2.67 $\pm$ 0.60	11.64 $\pm$ 4.58	6.59 $\pm$ 1.96	1.20 $\pm$ 0.46	0.71 $\pm$ 0.21
Ours	6.69 $\pm$ 2.76	2.23 $\pm$ 0.99	8.12 $\pm$ 1.73	2.51 $\pm$ 0.59	11.28 $\pm$ 4.65	6.40 $\pm$ 1.96	1.23 $\pm$ 0.41	0.81 $\pm$ 0.22
FCA	6.51 $\pm$ 2.45	2.17 $\pm$ 0.82	15.10 $\pm$ 4.06	2.95 $\pm$ 0.69	15.63 $\pm$ 7.18	5.99 $\pm$ 1.86	1.77 $\pm$ 0.57	0.67 $\pm$ 0.22
FCED	6.26 $\pm$ 2.35	2.14 $\pm$ 0.86	14.61 $\pm$ 3.95	2.75 $\pm$ 0.63	15.87 $\pm$ 7.73	5.94 $\pm$ 1.81	1.81 $\pm$ 0.71	0.73 $\pm$ 0.20

Table 6. Table S.4 : Average per-vertex errors on the test sets of DFaust (in c ​ m 𝑐 𝑚 cm ), SynHand5M (in m ​ m 𝑚 𝑚 mm ), textureless cloth (in c ​ m 𝑐 𝑚 cm ) and CoMA (in m ​ m 𝑚 𝑚 mm ). We compare two versions of DEMEA: one with spiral convolutions and one with spectral convolutions.

	DFaust		SynHand5M		Cloth		CoMA
	8	32	8	32	8	32	8	32
Spiral	6.69	2.23	8.12	2.51	11.28	6.40	1.23	0.81
Spectral	6.56	2.40	8.74	3.83	11.76	6.52	1.40	0.98

Table 7. Table S.5 : Qualitative Results on FCA.

Ground-truth	FCA 32	DEMEA 32	FCA 8	DEMEA 8

Table 8. Table S.6 : Artifacts on FCA. While the reconstruction by DEMEA only matches the ground-truth as well as FCA, it is a significantly more plausible shape.

Ground-truth	FCA 32	DEMEA 32

Table 9. Table S.7 : Qualitative Results on CA.

Ground-truth	CA 32	DEMEA 32	CA 8	DEMEA 8

Table 10. Table S.8 : Depth-to-Mesh Results for CA and FCA. The bottom row shows artifacts that DEMEA avoids.

Depth	DEMEA	CA	FCA

Table 11. Table S.9 : Coarse Embedded Graphs. Note the lips. An embedded graph on the second level of the mesh hierarchy instead of the first level can lead to over-smoothing.

Second Level	Ground-truth	First Level

Equations

$${\mathbf{T}}_l(\mathbf{p})={\mathbf{R}}_l[\mathbf{p}-{\mathbf{g}}_l]+{\mathbf{g}}_l+{\mathbf{t}}_l.$$

$$\mathbf{G}(\mathbf{p})=\sum _{l\in {\mathcal{N}}_{\mathbf{p}}}{w}_l(\mathbf{p})\cdot {\mathbf{T}}_l(\mathbf{p}).$$

$$w_l(\mathbf{p})=\exp \left(\frac{-\Vert {\mathbf{g}}_l-\mathbf{p}{\Vert }^2}{2\cdot {\sigma }^2}\right),$$

$$\sigma ={\sigma }_0\cdot d_{max}\cdot \frac{1}{\sqrt{L}},$$

$${\mathcal{L}}_{vertex}=\frac{1}{N_v}\sum _{i=1}^{N_v}\Vert {\hat{\mathbf{v}}}_i-{\mathbf{v}}_i^{\ast }{\Vert }_1,$$

$${\mathcal{L}}_{graph}=\frac{1}{L}\sum _{l=1}^L\Vert {\mathbf{t}}_l-{\mathbf{v}}_{i_l}^{\ast }{\Vert }_1,$$

$${\mathbf{y}}_{j,\ast }^T=\sum _{s=0}^{S-1}{G}_s\cdot {\mathbf{x}}_{n_s,\ast }^T,$$

$${\mathbf{y}}_{\ast ,j}=\sum _{i=1}^{F_{in}}{g}_{{\theta }_{i,j}}(\mathbf{L})\cdot {\mathbf{x}}_{\ast ,i}.$$

$$g_{{\theta }_{i,j}}(\mathbf{L})=\sum _{k=0}^{K-1}{\theta }_{i,j,k}\cdot T_k(\tilde{\mathbf{L}}),$$

Full text

1 Introduction

With the increasing volume of datasets of deforming objects enabled by modern 3D acquisition technology, the demand for compact data representations and compression grows. Dimensionality reduction of mesh data has multiple applications in computer graphics and vision, including shape retrieval, generation, interpolation, and completion. Recently, deep convolutional autoencoder networks were shown to produce compact mesh representations [2, 37, 31, 6].

Dynamic real-world objects do not deform arbitrarily. While deforming, they preserve topology, and nearby points are more likely to deform similarly compared to more distant points. Current convolutional mesh autoencoders exploit this coherence by learning the deformation properties of objects directly from data and are already suitable for mesh compression and representation learning. On the other hand, they do not explicitly reason about the deformation field in terms of local rotations and translations. We show that explicitly reasoning about the local rigidity of meshes enables higher-quality results for highly deformable objects, compared to directly regressing vertex positions.

At the other end of the spectrum, mesh manipulation techniques such as As-Rigid-As-Possible Deformation [34] and Embedded Deformation [35] only require a single mesh and enforce deformation properties, such as smoothness and local rigidity, based on a set of hand-crafted priors. These hand-crafted priors are effective and work surprisingly well, but since they do not model the real-world deformation behavior of the physical object, they often lead to unrealistic deformations and artifacts in the reconstructions.

In this paper, we propose a general-purpose mesh autoencoder with a model-based deformation layer, combining the best of both worlds, i.e., supervised learning with deformable meshes and a novel differentiable embedded deformation layer that models the deformable meshes using lower-dimensional deformation graphs with physically interpretable deformation parameters. While the core of our DEep MEsh Autoencoder (DEMEA) learns the deformation model of objects from data using the state-of-the-art convolutional mesh autoencoder (CoMA) [31], the novel embedded deformation layer decouples the parameterization of object motion from the mesh resolution and introduces local spatial coherence via vertex skinning. DEMEA is trained on mesh datasets of moderate sizes that have recently become available [24, 4, 3, 26]. DEMEA is a general mesh autoencoding approach that can be trained for any deformable object class. We evaluate our approach on datasets of three objects with large deformations like articulated deformations (body, hand) and large non-linear deformations (cloth), and one object with small localized deformations (face). Quantitatively, DEMEA outperforms standard convolutional mesh autoencoder architectures in terms of vertex-to-vertex distance error. Qualitatively, we show that DEMEA produces visually higher fidelity results due to the physically based embedded deformation layer. We show several applications of DEMEA in computer vision and graphics. Once trained, the decoder of our autoencoders can be used for shape compression, high-quality depth-to-mesh reconstruction of human bodies and hands, and even poorly textured RGB-image-to-mesh reconstruction for deforming cloth. The low-dimensional latent space learned by our approach is meaningful and well-behaved, which we demonstrate by different applications of latent space arithmetic. Thus, DEMEA provides us a well-behaved general-purpose category-specific generative model of highly deformable objects.

2 Related Work

Mesh Manipulation and Tracking. Our embedded deformation layer is inspired by as-rigid-as-possible modelling [34] and the method of Sumner et al. [35] for mesh editing and manipulation. While these methods have been shown to be very useful for mesh manipulation in computer graphics, to the best of our knowledge, this is the first time a model-based regularizer is used in a mesh autoencoder. Using a template for non-rigid object tracking from depth maps was extensively studied in the model-based setting [22, 41]. Recently, Litany et al. [23] demonstrated a neural network-based approach for the completion of human body shapes from a single depth map.

**Graph Convolutions. ** The encoder-decoder approach to dimensionality reduction with neural networks (NNs) for images was introduced in [17]. Deep convolutional neural networks (CNNs) allow to effectively capture contextual information of input data modalities and can be trained for various tasks. Lately, convolutions operating on regular grids have been generalized to more general topologically connected structures such as meshes and two-dimensional manifolds [7, 29], enabling learning of correspondences between shapes, shape retrieval [27, 5, 28], and segmentation [40]. Masci et al. [27] proposed geodesic CNNs operating on Riemannian manifolds for shape description, retrieval, and correspondence estimation. Boscani et al. [5] introduced spatial weighting functions based on simulated heat propagation and projected anisotropic convolutions. Monti et al. [28] extended graph convolutions to variable patches through Gaussian mixture model CNNs. In FeaSTNet [38], the correspondences between filter weights and graph neighborhoods with arbitrary connectivities are established dynamically from the learned features. The localized spectral interpretation of Defferrard et al. [9] is based on recursive feature learning with Chebyshev polynomials and has linear evaluation complexity. Focusing on mesh autoencoding, Bouritsas et al. [6] exploited the fixed ordering of neighboring vertices.

**Learning Mesh-Based 3D Autoencoders. ** Very recently, several mesh autoencoders with various applications were proposed. A new hierarchical variational mesh autoencoder with fully connected layers for facial geometry parameterization learns an accurate face model from small databases and accomplishes depth-to-mesh fitting tasks [2]. Tan and coworkers [36] introduced a mesh autoencoder with a rotation-invariant mesh representation as a generative model. Their network can generate new meshes by sampling in the latent space and perform mesh interpolation. To cope with meshes of arbitrary connectivity, they used fully-connected layers and did not explicitly encode neighbor relations. Tan et al. [37] trained a network with graph convolutions to extract sparse localized deformation components from meshes. Their method is suitable for large-scale deformations and meshes with irregular connectivity. Gao et al. [12] transferred mesh deformations by training a generative adversarial network with a cycle consistency loss to map shapes in the latent space, while a variational mesh autoencoder encodes deformations. The Convolutional facial Mesh Autoencoder (CoMA) of Ranjan et al. [31] allows to model and sample stronger deformations compared to previous methods and supports asymmetric facial expressions. The Neural 3DMM of Bouritsas et al. [6] improves quantitatively over CoMA due to better training parameters and task-specific graph convolutions. Similar to CoMA [31], our DEMEA uses spectral graph convolutions but additionally employs the embedded deformation layer as a model-based regularizer.

**Learning 3D Reconstruction. ** Several supervised methods reconstruct rigid objects in 3D. Given a depth image, the network of Sinha et al. [33] reconstructs the observed surface of non-rigid objects. In its 3D reconstruction mode, their method reconstructs rigid objects from single images. Similarly, Groueix et al. [15] reconstructed object surfaces from a point cloud or single monocular image with an atlas parameterization. The approaches of Kurenkov et al. [21] and Jack et al. [18] deform a predefined object-class template to match the observed object appearance in an image. Similarly, Kanazawa et al. [19] deformed a template to match the object appearance but additionally support object texture. The Pixel2Mesh approach of Wang et al. [39] reconstructs an accurate mesh of an object in a segmented image. Initializing the 3D reconstruction with an ellipsoid, their method gradually deforms it until the appearance matches the observation. The template-based approaches [21, 18, 19], as well as Pixel2Mesh [39], produce complete 3D meshes.

**Learning Monocular Non-Rigid Surface Regression. ** Only a few supervised learning approaches for 3D reconstruction from monocular images tackle the deformable nature of non-rigid objects. Several methods [30, 14, 32] train networks for deformation models with synthetic thin plates datasets. These approaches can infer non-rigid states of the observed surfaces such as paper sheets or membranes. Still, their accuracy and robustness on real images are limited. Bednařík et al. [3] proposed an encoder-decoder network for texture-less surfaces relying on shading cues. They trained on a real dataset and showed an enhanced reconstruction accuracy on real images, but support only trained object classes. Fuentes-Jimenez et al. [11] trained a network to deform an object template for depth map recovery. They achieved impressive results on real image sequences but require an accurate 3D model of every object in the scene, which restricts the method’s practicality. One of the applications of DEMEA is the recovery of texture-less surfaces from RGB images. Since a depth map as a data modality is closer to images with shaded surfaces, we train DEMEA in the depth-to-mesh mode on images instead of depth maps. As a result, we can regress surface geometry from shading cue.

3 Approach

In this section, we describe the architecture of the proposed DEMEA. We employ an embedded deformation layer to decouple the complexity of the learned deformation field from the actual mesh resolution. The deformation is represented relative to a canonical mesh $\mathcal{M}=(\mathbf{V},\mathbf{E})$ with $N_{v}$ vertices $\mathbf{V}=\{\mathbf{v}_{i}\}_{i=1}^{N_{v}}$, and edges $\mathbf{E}$. To this end, we define the encoder-decoder on a coarse deformation graph and use the embedded deformation layer to drive the deformation of the final high-resolution mesh, see Fig. S.1. Our architecture is based on graph convolutions that are defined on a multi-resolution mesh hierarchy. In the following, we describe all components in more detail. We describe the employed spiral graph convolutions [6] in the supplemental document.

3.1 Mesh Hierarchy

The up- and downsampling in the convolutional mesh autoencoder is defined over a multi-resolution mesh hierarchy, similar to the CoMA [31] architecture. We compute the mesh hierarchy fully automatically based on quadric edge collapses [13], i.e., each hierarchy level is a simplified version of the input mesh. We employ a hierarchy with five resolution levels, where the finest level is the mesh. Given the multi-resolution mesh hierarchy, we define up- and downsampling operations [31] for feature maps defined on the graph. To this end, during downsampling, we enforce the nodes of the coarser level to be a subset of the nodes of the next finer level. We transfer a feature map to the next coarser level by a similar subsampling operation. The inverse operation, i.e., feature map upsampling, is implemented based on a barycentric interpolation of close features. During edge collapse, we project each collapsed node onto the closest triangle of the coarser level. We use the barycentric coordinates of this closest point with respect to the triangle’s vertices to define the interpolation weights.

3.2 Embedded Deformation Layer (EDL)

Given a canonical mesh, we have to pick a corresponding coarse embedded deformation graph. We employ MeshLab’s [8] more sophisticated implementation of quadric edge collapse to fully automatically generate the graph. See the supplemental document for details.

The deformation graph is used as one of the two levels immediately below the mesh in the mesh hierarchy (depending on the resolution of the graph) of the autoencoder. When generating the mesh hierarchy, we need to enforce the subset relationship between levels. However, the quadric edge collapse algorithm of [31] might delete nodes of the embedded graph when computing intermediate levels between the mesh and the embedded graph. We ensure that those nodes are not removed by setting the cost of removing them from levels that are at least as fine as the embedded graph to infinity.

Our embedded deformation layer models a space deformation that maps the vertices of the canonical template mesh $\mathbf{V}$ to a deformed version $\hat{\mathbf{V}}$. Suppose $\mathcal{G}=(\mathbf{N},\mathbf{E})$ is the embedded deformation graph [35] with $L$ canonical nodes $\mathbf{N}=\{{\boldsymbol{g}}_{l}\}_{i=1}^{L}$ and $K$ edges $\mathbf{E}$, with ${\boldsymbol{g}}_{l}\in\mathbb{R}^{3}$. The global space deformation is defined by a set of local, rigid, per-graph node transformations. Each local rigid space transformation is defined by a tuple ${\boldsymbol{T}}_{l}=({\boldsymbol{R}}_{l},{\boldsymbol{t}}_{l})$, with ${\boldsymbol{R}}_{l}\in\mathbf{SO}(3)$ being a rotation matrix and ${\boldsymbol{t}}_{l}\in\mathbb{R}^{3}$ being a translation vector. We enforce that ${\boldsymbol{R}}_{l}^{\mathsf{T}}={\boldsymbol{R}}_{l}^{-1}$ and $\operatorname{det}({\boldsymbol{R}}_{l})=1$ by parameterizing the rotation matrices based on three Euler angles. Each ${\boldsymbol{T}}_{l}$ is anchored at the canonical node position ${\boldsymbol{g}}_{l}$ and maps every point ${\boldsymbol{p}}\in\mathbb{R}^{3}$ to a new position in the following manner [35]:

[TABLE]

To obtain the final global space deformation $\mathbf{G}$, the local per-node transformations are linearly combined:

[TABLE]

Here, $\mathcal{N}_{\boldsymbol{p}}$ is the set of approximate closest deformation nodes. The linear blending weights ${w_{l}}({\boldsymbol{p}})$ for each position are based on the distance to the respective deformation node [35]. Please refer to the supplemental for more details.

The deformed mesh $\hat{\mathbf{V}}=\mathbf{G}(\mathbf{V})$ is obtained by applying the global space deformation to the canonical template mesh $\mathbf{V}$. The free parameters are the local per-node rotations ${\boldsymbol{R}}_{l}$ and translations ${\boldsymbol{t}}_{l}$, i.e., $6L$ parameters with $L$ being the number of nodes in the graph. These parameters are input to our deformation layer and are regressed by the graph convolutional decoder.

3.3 Differentiable Space Deformation

Our novel EDL is fully differentiable and can be used during network training to decouple the parameterization of the space deformation from the resolution of the final high-resolution output mesh. This enables us to define the reconstruction loss on the final high-resolution output mesh and backpropagate the errors via the skinning transform to the coarse parameterization of the space deformation. Thus, our approach enables finding the best space deformation by only supervising the final output mesh.

3.4 Training

We train our approach end-to-end in Tensorflow [1] using Adam [20]. As loss we employ a dense geometric per-vertex $\ell_{1}$-loss with respect to the ground-truth mesh. For all experiments, we use a learning rate of $10^{-4}$ and default parameters $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$ for Adam. We train for $50$ epochs for Dynamic Faust, $30$ epochs for SynHand5M, $50$ epochs for the CoMA dataset and $300$ epochs for the Cloth dataset. We employ a batch size of $8$.

3.5 Reconstructing Meshes from Images/Depth

The image/depth-to-mesh network consists of an image encoder and a mesh decoder, see Fig. 5(a). The mesh decoder is initialized from the corresponding mesh auto-encoder, the image/depth encoder is based on a ResNet-50 [16] architecture, and the latent code is shared between the encoder and decoder. We initialize the ResNet-50 component using pre-trained weights from ImageNet [10]. To obtain training data, we render synthetic depth maps from the meshes. We train with the same settings as for mesh auto-encoding.

3.6 Network Architecture Details

In the following, we provide more details of our encoder-decoder architectures.

Encoding Meshes. Input to the first layer of our mesh encoder is an $N_{v}\times 3$ tensor that stacks the coordinates of all $N_{v}$ vertices. We apply four downsampling modules. Each module applies a graph convolution and is followed by a downsampling to the next coarser level of the mesh hierarchy. We use spiral graph convolutions [6] and similarly apply an ELU non-linearity after each convolution. Finally, we take the output of the final module and apply a fully connected layer followed by an ELU non-linearity to obtain a latent space embedding.

Encoding Images/Depth. To encode images/depth, we employ a 2D convolutional network to map color/depth input to a latent space embedding. Input to our encoder are images of resolution $256\times 256$ pixels. We modified the ResNet-50 [16] architecture to take single or three-channel input image. We furthermore added two additional convolution layers at the end, which are followed by global average pooling. Finally, a fully connected layer with a subsequent ELU non-linearity maps the activations to the latent space.

Decoding Graphs. The task of the graph decoder is to map from the latent space back to the embedded deformation graph. First, we employ a fully connected layer in combination with reshaping to obtain the input to the graph convolutional upsampling modules. We apply a sequence of three or four upsampling modules until the resolution level of the embedded graph is reached. Each upsampling module first up-samples the features to the next finer graph resolution and then performs a graph convolution, which is then followed by an ELU non-linearity. Then, we apply two graph convolutions with ELUs for refinement and a final convolution without an activation function. The resulting tensor is passed to our embedded deformation layer.

4 Experiments

We evaluate DEMEA quantitatively and qualitatively on several challenging datasets and demonstrate state-of-the-art results for mesh auto-encoding. In Sec. 5, we show reconstruction from RGB images and depth maps and that the learned latent space enables well-behaved latent arithmetic. We use Tensorflow 1.5.0 [1] on Debian with an NVIDIA Tesla V100 GPU.

Datasets. We demonstrate DEMEA’s generality on experiments with body (Dynamic Faust, DFaust [4]), hand (SynHand5M [26]), textureless cloth (Cloth [3]), and face (CoMA [31]) datasets.

Table 2 gives the number of graph nodes used on each level of our hierarchical architecture. All meshes live in metric space.

DFaust [4]. The training set consists of 28,294 meshes. For the tests, we split off two identities (female 50004, male 50002) and two dynamic performances, i.e., one-leg jump and chicken wings. Overall, this results in a test set with $12,926$ elements. For the depth-to-mesh results, we found the synthetic depth maps from the DFaust training set to be insufficient for generalization, i.e., the test error was high. Thus, we add more pose variety to DFaust for the depth-to-mesh experiments. Specifically, we add $28k$ randomly sampled poses from the CMU Mocap111mocap.cs.cmu.edu dataset to the training data, where the identities are randomly sampled from the SMPL [25] model (14$k$ female, 14$k$ male). We also add 12$k$ such samples to the test set (6$k$ female, 6$k$ male).

Textureless Cloth [3]. For evaluating our approach on general non-rigidly deforming surfaces, we use the textureless cloth data set of Bednařík et al. [3]. It contains real depth maps and images of a white deformable sheet — observed in different states and differently shaded — as well as ground-truth meshes. In total, we select 3,861 meshes with consistent edge lengths. 3,167 meshes are used for training and $700$ meshes are reserved for evaluation. Since the canonical mesh is a perfectly flat sheet, it lacks geometric features, which causes downsampling methods like [13], [31] and [8] to introduce severe artifacts. Hence, we generate the entire mesh hierarchy for this dataset, see the supplemental. This hierarchy is also used to train the other methods for the performed comparisons.

SynHand5M [26]. For the experiments with hands, we take $100k$ random meshes from the synthetic SynHand5M dataset of Malik et al. [26]. We render the corresponding depth maps. The training set is comprised of $90k$ meshes, and the remaining $10k$ meshes are used for evaluation.

CoMA [31]. The training set contains 17,794 meshes of the human face in various expressions [31]. For tests, we select two challenging expressions, i.e., high smile and mouth extreme. Thus, our test set contains 2,671 meshes in total.

4.1 Baseline Architectures

We compare DEMEA to a number of strong baselines.

**Convolutional Baseline. ** We consider a version of our proposed architecture, convolutional ablation (CA), where the ED layer is replaced by learned upsampling modules that upsample to the mesh resolution. In this case, the extra refinement convolutions occur on the level of the embedded graph. We also consider modified CA (MCA), an architecture where the refinement convolutions are moved to the end of the network, such that they operate on mesh resolution.

Fully-Connected Baseline. We also consider an almost-linear baseline, FC ablation (FCA). The input is given to a fully-connected layer, after which an ELU is applied. The resulting latent vector is decoded using another FC layer that maps to the output space. Finally, we also consider an FCED network where the fully-connected decoder maps to the deformation graph, which the embedded deformation layer (EDL) in turn maps to the full-resolution mesh.

4.2 Evaluation Settings

We first determine how to integrate the EDL into the training. Our proposed architecture regresses node positions and rotations and then uses the EDL to obtain the deformed mesh, on which the reconstruction loss is applied.

As an alternative, we consider the graph loss (GL) with the $\ell_{1}$ reconstruction loss directly on the graph node positions (where the vertex positions of the input mesh that correspond to the graph nodes are used as ground-truth). The GL setting uses the EDL only at test time to map to the full mesh, but not for training. Although the trained network predicts graph node positions $\boldsymbol{t}_{l}$ at test time, it does not regress graph node rotations $\boldsymbol{R}_{l}$ which are necessary for the EDL. We compute the missing rotation for each graph node $l$ as follows: assuming that each node’s neighborhood transforms roughly rigidly, we solve a small Procrustes problem that computes the rigid rotation between the 1-ring neighborhoods of $l$ in the template graph and in the regressed network output. We directly use this rotation as $\boldsymbol{R}_{l}$.

We also consider the alternative of estimating the local Procrustes rotation inside the network during training (LP). We add a reconstruction loss on the deformed mesh as computed by the EDL. Here, we do not back-propagate through the rotation computation to avoid training instabilities.

Table 4 shows the quantitative results using the average per-vertex Euclidean error. Using the EDL during training leads to better quantitative results, as the network is aware of the skinning function and can move the graph nodes accordingly. In addition to being an order of magnitude faster than LP, regressing rotations either gives the best results or is close to the best. We use the EDL with regressed rotation parameters during training in all further experiments.

We use spiral graph convolutions [6], but show in the supplemental document that spectral graph convolutions [9] also give similar results.

4.3 Evaluations of the Autoencoder

Qualitative Evaluations. Our architecture significantly outperforms the baselines qualitatively on the DFaust and SynHand5M datasets, as seen in Figs. S.1 and 4. Convolutional architectures without an embedded graph produce strong artifacts in the hand, feet and face regions in the presence of large deformations. Since EDL explicitly models deformations, we preserve fine details of the template under strong non-linear deformations and articulations of extremities.

**Quantitative Evaluations. ** We compare the proposed DEMEA to the baselines on the autoencoding task, see Table 2.

While the fully-connected baselines are competitive for larger dimensions of the latent space, their memory demand increases drastically. On the other hand, they perform significantly worse for low dimensions on all datasets, except for DFaust. In this work, we are interested in low latent dimensions, e.g. less than 32, as we want to learn mesh representations that are as compact as possible. We also observe that adding EDL to the fully-connected baselines maintains their performance. Furthermore, the lower test errors of FCED on Cloth indicate that network capacity (and not EDL) limits the quantitative results.

On SynHand5M, Cloth and CoMA, the convolutional baselines perform on par with DEMEA. On DFaust, our performance is slightly worse, perhaps because other architectures can also fit to the high-frequency details and noise. EDL regularizes deformations to avoid artifacts, which also prevents fitting to high-frequency or small details. Thus, explicitly modelling deformations via the EDL and thereby avoiding artifacts has no negative impact on the quantitative performance. Since CoMA mainly contains small and local deformations, DEMEA does not lead to any quantitative improvement. This is more evident in the case of latent dimension 32, as the baselines can better reproduce noise and other high-frequency deformations.

Comparisons. In extensive comparisons with several competitive baselines, we have demonstrated the usefulness of our approach for autoencoding strong non-linear deformations and articulated motion. Next, we compare DEMEA to the existing state-of-the-art CoMA approach [31]. We train their architecture on all mentioned datasets with a latent dimension of $8$, which is also used in [31]. We outperform their method quantitatively on DFaust ($6.7cm$ vs. $8.4cm$), on SynHand5M ($8.12mm$ vs. $8.93mm$), on Cloth ($1.13cm$ vs. $1.44cm$), and even on CoMA ($1.23mm$ vs. $1.42mm$), where the deformations are not large. We also compare to Neural 3DMM [6] on latent dimensions $8$ and $32$, similarly to [31] on their proposed hierarchy. See Table 4 for the results. DEMEA performs better than Neural 3DMM in almost all cases. In Fig. S.1, we show that DEMEA avoids many of the artifacts present in the case of [31], [6] and other baselines.

5 Applications

5.1 RGB to Mesh

On the Cloth [3] dataset, we show that DEMEA can reconstruct meshes from RGB images. See Fig. 5(b) for qualitative examples with a latent dimension of $32$.

On our test set, our proposed architecture achieves RGB-to-mesh reconstruction errors of $16.1mm$ and $14.5mm$ for latent dimensions $8$ and $32$, respectively. Bednařík et al. [3], who use a different split than us, report an error of $21.48mm$. The authors of IsMo-GAN [32] report results on their own split for IsMo-GAN and the Hybrid Deformation Model Network (HDM-net) [14]. On their split, HDM-Net achieves an error of $17.65mm$ after training for 100 epochs using a batch size of 4. IsMo-GAN obtains an error of $15.79mm$. Under the same settings as HDM-Net, we re-train our approach without pre-training the mesh decoder. Our approach achieves test errors of $16.6mm$ and $13.8mm$ using latent dimensions of $8$ and $32$, respectively.

5.2 Depth to Mesh

Bodies. We train a network with a latent space dimension of $32$. Quantitatively, we obtain an error of $2.3cm$ on un-augmented synthetic data. Besides, we also apply our approach to real data, see Fig. 6(a).

To this end, we found it necessary to augment the depth images with artificial noise to lessen the domain gap. Video results are included in the supplementary.

**Hands. ** DEMEA can reconstruct hands from depth as well, see Fig. 6(b). We achieve a reconstruction error of $6.73mm$ for a latent dimension of $32$. Malik et al. [26] report an error of $11.8$ $mm$. Our test set is composed of a random sample of fully randomly generated hands from the dataset, which is very challenging. We use $256\times 256$, whereas [26] use images of size $96\times 96$.

5.3 Latent Space Arithmetic

Although we do not employ any regularization on the latent space, we found empirically that the network learns a well-behaved latent space. As we show in the supplemental document and video, this allows DEMEA to temporally smooth tracked meshes from a depth stream.

Latent Interpolation. We can linearly interpolate the latent vectors $\mathcal{S}$ and $\mathcal{T}$ of a source and a target mesh: $\mathcal{I}(\alpha)=(1-\alpha)\mathcal{S}+\alpha\mathcal{T}$. Even for highly different poses and identities, these $\mathcal{I}(\alpha)$ yield plausible in-between meshes, see Fig. 7(a).

Deformation Transfer. The learned latent space allows to transfer poses between different identities on DFaust. Let a sequence of source meshes $\mathbf{S}=\{\textbf{M}_{i}\}_{i}$ of person $A$ and a target mesh $\textbf{M}^{\prime}_{0}$ of person $B$ be given, where w.l.o.g. $\textbf{M}_{0}$ and $\textbf{M}^{\prime}_{0}$ correspond to the same pose. We now seek a sequence of target meshes $\mathbf{S}^{\prime}=\{\textbf{M}^{\prime}_{i}\}_{i}$ of person $B$ performing the same poses as person $A$ in $\mathbf{S}$. We encode $\mathbf{S}$ and $M^{\prime}_{0}$ into the latent space of the mesh auto-encoder, yielding the corresponding latent vectors $\{\mathcal{M}_{i}\}_{i}$ and $\mathcal{M}^{\prime}_{0}$. We define the identity difference $d=\mathcal{M}^{\prime}_{0}-\mathcal{M}_{0}$ and set $\mathcal{M}^{\prime}_{i}=\mathcal{M}_{i}+d$ for $i>0$. Decoding $\{\mathcal{M}^{\prime}_{i}\}_{i}$ using the mesh decoder than yields $\mathbf{S^{\prime}}$. See Fig. 7(b) and the supplement for qualitative results.

6 Limitations

While the embedded deformation graph excels on highly articulated, non-rigid motions, it has difficulties accounting for very subtle actions. Since the faces in the CoMA [31] dataset do not undergo large deformations, our EDL-based architecture does not offer a significant advantage. Similar to all other 3D deep learning techniques, our approach also requires reasonably sized mesh datasets for supervised training, which might be difficult to capture or model. We train our network in an object-specific manner. Generalizing our approach across different object categories is an interesting direction for future work.

7 Conclusion

We proposed DEMEA — the first deep mesh autoencoder for highly deformable and articulated scenes, such as human bodies, hands, and deformable surfaces, that builds on a new differentiable embedded deformation layer. The deformation layer reasons about local rigidity of the mesh and allows us to achieve higher quality autoencoding results compared to several baselines and existing approaches. We have shown multiple applications of our architecture including non-rigid reconstruction from real depth maps and 3D reconstruction of textureless surfaces from images.

Acknowledgments. This work was supported by the ERC Consolidator Grant 4DReply (770784), the Max Planck Center for Visual Computing and Communications (MPC-VCC), and an Oculus research grant.

S.1 Depth-to-Mesh Tracking

We can apply temporal smoothing to the reconstruction of a sequence of real depth images $\{\textbf{D}_{i}\}_{i}$, by decoding a running (causal) exponential average of the latent vectors of this sequence. First, we encode the sequence into latent vectors $\{\mathcal{D}_{i}\}_{i}$. We then define a smoothed sequence of latent vectors $\{\mathcal{D}^{\prime}_{i}\}_{i}$ as follows: let $\mathcal{D}^{\prime}_{0}=\mathcal{D}_{0}$ and set $\mathcal{D}^{\prime}_{i}=\alpha\cdot\mathcal{D}_{i}+(1-\alpha)\cdot\mathcal{D}^{\prime}_{i-1}$ for $i>0$ for some $\alpha\in[0,1]$. The smoothed sequence of meshes $\{\textbf{M}_{i}\}_{i}$ is obtained by decoding $\{\mathcal{D}^{\prime}_{i}\}_{i}$. See the supplemental video for examples.

S.2 Skinning

We compute the linear blending weight $w_{l}({\boldsymbol{p}})$ as [35]:

[TABLE]

where we determine $\sigma\in\mathbb{R}$ heuristically as follows:

[TABLE]

where $\sigma_{0}=\frac{2}{3}$ and $d_{max}$ is the maximum Euclidean distance between any two mesh vertices, a proxy for the absolute scale of the mesh.

Note that $\mathcal{N}_{\boldsymbol{p}}$, ${w_{l}}$ and $\sigma$ are pre-computed on the template mesh and graph and are kept fixed within each dataset. We use $\lvert\mathcal{N}_{{\boldsymbol{p}}}\rvert=6$ for embedded graphs on the first level and $\lvert\mathcal{N}_{{\boldsymbol{p}}}\rvert=12$ for embedded graphs on the second level.

S.3 Artifacts

Tables S.1 and S.2 show additional examples of artifacts that occur when not integrating the embedded graph into the network. Even the most competitive network against which we compare (*i.e. *our ablation) suffers from visually unpleasing artifacts due to large non-rigid deformations, which most visibly occur on the hands and feet of DFaust. However, due to the localized nature of the artifacts, they do not have a large impact on the quantitative errors.

S.4 Architecture

Fig. S.1 contains our low-level architecture.

GC(f) is a graph-convolutional layer with $f$ output features. DS is a downsampling layer and US is an upsampling layer. Conv2D(f,k,s) is a 2D convolution with $f$ output features, kernel size $k\times k$ and stride $s$. We modified ResNetV2 50 by removing its first convolutional layer and its final non-convolutional layers, and keep its architecture unmodified otherwise. We use ELU non-linearities after every graph-convolutional, 2D convolutional and fully-connected layer except for the first 2D convolutional layer in the depth encoder and the last graph-convolutional layer. The third upsampling module (i.e. upsampling layer followed by a graph convolution) is only used for higher-resolution embedded graphs. In the case of spiral graph convolutions, we use the default settings of [6] to determine the length of the spirals. In the case of spectral graph convolutions, we always use $K=6$, except for the last two layers, which use $K=2$ for local refinement.

S.5 Losses

The vertex loss mentioned in Sec. 3.4 is:

[TABLE]

where $\mathbf{\hat{v}}_{i}$ is the $i$-th deformed vertex and $\mathbf{v}^{*}_{i}$ is the $i$-th ground-truth vertex. The loss is averaged across the batch.

The deformed vertex can be either directly regressed (CA, MCA, FCA) or it can be computed via EDL (DEMEA, FCED). In the latter case, we follow Eq. 2 from the main manuscript. While DEMEA regresses all EDL parameters, we also try out a modification that instead uses local Procrustes inside the network, called LP, see Sec. 3.4. In that case, we use the same loss (Eq. 5) as for DEMEA but only regress the translation parameters, $\boldsymbol{t}_{l}$, while computing the rotation parameters of EDL, $\boldsymbol{R}_{l}$, using local Procrustes as described in Sec. 3.4. Note that we do not backpropagate through the rotation computation in this case.

The graph loss described in Sec. 3.4 of the main manuscript operates on the graph nodes. It encourages the regressed graph nodes positions to be close to the ground-truth vertex positions. The graph nodes $\mathbf{N}$ are a subset of the mesh vertices $\mathbf{V}$ and we denote the vertex index corresponding to a graph node $l$ as $i_{l}$. Then, the graph loss is:

[TABLE]

S.6 Normalization

Depth

All depth-to-mesh networks rescale the depth values of the input depth map from between $0.3m$ and $7m$ to $[-1,1]$.

Bodies: Depth

For our depth-to-mesh network on bodies, we employ a number of additional normalization steps to focus on non-rigid reconstruction. First, we assume to be given a segmentation mask that filters out the background. The depth value of background pixels is set to 2. We crop the foreground tightly and use bilinear sampling to isotropically rescale the crop to $256\times 256$. Given such a depth crop, we compute the average (foreground) depth value and subtract it from the input. Such normalization necessitates normalizing the network output, as we will describe next.

Bodies: Meshes

We first normalize out the global translation from the meshes by subtracting from each mesh vertex the average vertex position. Since scale information is also lost, we fix the scale of the meshes by normalizing their approximate spine length. To that end, we compute the approximate spine length of the template mesh and of each mesh in the dataset. We then isotropically rescale all the meshes to the same spine length as the template mesh. The depth-to-mesh body reconstruction errors in the main paper are reported for these normalized meshes.

S.7 Standard Deviations in Table 2

Due to space reasons, we could not fit standard deviations across the test set in Table 2 of the main manuscript. Table S.3 contains the expanded version.

S.8 Mesh Hierarchy

We use the code of [31] to generate the mesh hierarchy. Then, in practice, the first or the second level of this automatically generated mesh hierarchy can be used as the embedded graph.

However, we propose to use more uniform embedded graphs. Using MeshLab’s [8] implementation of quadric edge collapse decimation with default settings works well. (Note that we decimate the mesh until we reach the same number of graph nodes as used by [31].) We experimented with different ways of obtaining better embedded graphs, including hand-crafting them, but found no major differences, except that graphs generated by [31] were too non-uniform. Among the tested methods, MeshLab constitutes the least involved method of obtaining a uniform graph.

Since the embedded graph needs to be a subset of the mesh, graph nodes obtained this way need to be projected to their closest vertices. This may lead to multiple nodes projecting to the same vertex. We resolve this with a greedy assignment from nodes to vertices: looping over all nodes, the current node is assigned to its closest vertex that is not yet taken by another node. With the modification of the code of [31] described in the main paper, we can then generate the mesh hierarchy for this embedded graph.

Figure S.2 shows the hierarchy we use for Cloth.

S.9 Graph Convolutions

Our graph encoder-decoder architecture can work with multiple types of graph convolutions, which we show in this section. The results of the main paper use spiral graph convolutions, as we found these to give slightly better results than spectral graph convolutions. After defining the two types of graph convolutions, we report quantitative results.

Given an $F_{in}$-channel feature tensor $\mathbf{x}\in\mathbb{R}^{N\times F_{in}}$, where the features are defined at the $N$ graph nodes, let $\mathbf{x}_{*,i}\in\mathbb{R}^{N}$ denote the $i$-th input graph feature map. The complete output feature tensor, that stacks all $F_{out}$ feature maps, is denoted as $\mathbf{y}\in\mathbb{R}^{N\times F_{out}}$. We apply the graph convolutions without stride, i.e., the input graph resolution equals the output resolution.

S.9.1 Spiral Graph Convolutions

We mainly work with spiral graph convolutions [6]. Let $\mathbf{x}_{n,*}^{T}\in\mathbb{R}^{F_{in}}$ denote the feature vector of graph node $n$. Assuming a fixed topology of the graph, we may choose an ordering of the neighboring nodes of graph node $n$: $n_{0},\ldots,n_{s},\ldots,n_{S-1}$. Bouritsas et al. pick a spiral ordering that starts with $n_{0}=n$ and proceeds along the 1-ring ($n^{1}_{0},n^{1}_{1},\ldots$), then the 2-ring ($n^{2}_{0},n^{2}_{1},\ldots$), and so on. The spiral ordering is thus given by: $n,n^{1}_{0},n^{1}_{1},\ldots,n^{2}_{0},n^{2}_{1},\ldots$ All the spirals of the graph are ultimately cut to a fixed length $S$ and zero-padded if necessary. The output of the spiral convolution is then defined as:

[TABLE]

where $G_{s}\in F_{out}\times F_{in}$ is a trainable matrix.

S.9.2 Spectral Graph Convolutions

The second type of graph convolutions is based on fast localized spectral filtering [9], which Ranjan et al. use in CoMA [31]. We compute the $j$-th output graph feature map $\mathbf{y}_{*,j}\in\mathbb{R}^{N}$ as follows:

[TABLE]

Here, $\mathbf{L}$ is the normalized Laplacian matrix of the graph and the filters $g_{\theta_{i,j}}(\mathbf{L})$ are parameterized using Chebyshev polynomials of order $K$. More specifically,

[TABLE]

where $\theta_{i,j,k}\in\mathbb{R}$ and $\mathbf{\widetilde{L}}=2\mathbf{L}/\lambda_{\textit{max}}-\mathbf{I}$, with $\lambda_{\textit{max}}=2$ being an upper bound on the spectrum of $\mathbf{L}$. The Chebychev polynomial $T_{k}$ is defined as $T_{k}(x)=2x\cdot T_{k-1}(x)-T_{k-2}(x)$, $T_{1}(x)=x$, and $T_{0}(x)=1$.

This leads to $K$-localized filters that operate on the $K$-neighbourhoods of the nodes. Each filter $g_{\theta_{i,j}}(\mathbf{L})$ is parameterized by $K$ coefficients, which in total leads to $F_{in}\times F_{out}\times K$ trainable parameters for each graph convolution layer.

S.9.3 Results

We compare our proposed DEMEA with spiral convolutions against a version of DEMEA that uses spectral convolutions. Table S.4 contains the results. Except for DFaust on latent dimension 8, spiral graph convolutions always perform at least slightly better than spectral graph convolutions. These results show that EDL obtains similar accuracy with both graph convolutions, which further validates its robustness.

S.10 FCA and CA Results

Fig. S.5 contains qualitative results for FCA and Fig. S.6 shows artifacts when using FCA. Fig. S.7 contains qualitative results for CA. We show depth-to-mesh results in Fig. S.8.

S.11 Coarse Embedded Graphs

In Fig. S.9, we show how an embedded graph can lead to over-smoothing and a loss of detail.