U-mesh: Human Correspondence Matching with Mesh Convolutional Networks

TL;DR

This paper introduces U-mesh, a mesh convolutional network that predicts human surface correspondence, combining regression and generative methods to improve accuracy on complex 3D scans with occlusions and partialities.

Contribution

It presents an intrinsic mesh U-net architecture for correspondence prediction and a generative optimization algorithm leveraging domain-specific priors.

Findings

Achieves 20% improvement on FAUST correspondence challenge

Handles occlusions, partial scans, and varying genus effectively

Combines mesh convolutional networks with generative model fitting

Abstract

The proliferation of 3D scanning technology has driven a need for methods to interpret geometric data, particularly for human subjects. In this paper we propose an elegant fusion of regression (bottom-up) and generative (top-down) methods to fit a parametric template model to raw scan meshes. Our first major contribution is an intrinsic convolutional mesh U-net architecture that predicts pointwise correspondence to a template surface. Soft-correspondence is formulated as coordinates in a newly-constructed Cartesian space. Modeling correspondence as Euclidean proximity enables efficient optimization, both for network training and for the next step of the algorithm. Our second contribution is a generative optimization algorithm that uses the U-net correspondence predictions to guide a parametric Iterative Closest Point registration. By employing pre-trained human surface parametric models we maximally leverage domain-specific prior knowledge. The pairing of a mesh-convolutional network with generative model fitting enables us to predict correspondence for real human surface scans including occlusions, partialities, and varying genus (e.g. from self-contact). We evaluate the proposed method on the FAUST correspondence challenge where we achieve 20% (33%) improvement over state of the art methods for inter- (intra-) subject correspondence.