Building 3D Morphable Models from a Single Scan

TL;DR

This paper introduces a novel method for creating 3D morphable models from a single scan, enabling shape and albedo modeling, inverse rendering, and recognition without needing multiple high-quality scans.

Contribution

It presents a new approach to build 3D morphable models from just one scan, expanding applicability to categories lacking dense correspondence.

Findings

Models can infer 3D from 2D and 3D data.

Single-scan models enable face recognition.

Models for fish and birds demonstrate versatility.

Abstract

We propose a method for constructing generative models of 3D objects from a single 3D mesh. Our method produces a 3D morphable model that represents shape and albedo in terms of Gaussian processes. We define the shape deformations in physical (3D) space and the albedo deformations as a combination of physical-space and color-space deformations. Whereas previous approaches have typically built 3D morphable models from multiple high-quality 3D scans through principal component analysis, we build 3D morphable models from a single scan or template. As we demonstrate in the face domain, these models can be used to infer 3D reconstructions from 2D data (inverse graphics) or 3D data (registration). Specifically, we show that our approach can be used to perform face recognition using only a single 3D scan (one scan total, not one per person), and further demonstrate how multiple scans can be incorporated to improve performance without requiring dense correspondence. Our approach enables the synthesis of 3D morphable models for 3D object categories where dense correspondence between multiple scans is unavailable. We demonstrate this by constructing additional 3D morphable models for fish and birds and use them to perform simple inverse rendering tasks. We share the code used to generate these models and to perform our inverse rendering and registration experiments.