3D Morphable Models as Spatial Transformer Networks

TL;DR

This paper introduces a novel neural network module that integrates 3D Morphable Models to improve spatial normalization of 3D pose, occlusion, and pose variations in images, enhancing robustness in face analysis tasks.

Contribution

It extends spatial transformer networks by incorporating 3D Morphable Models, enabling explicit 3D pose normalization and occlusion handling within CNNs.

Findings

Localiser learns to fit 3DMM to images with simple geometric loss

Robust normalization on images with occlusion and pose variation

Effective training on small datasets

Abstract

In this paper, we show how a 3D Morphable Model (i.e. a statistical model of the 3D shape of a class of objects such as faces) can be used to spatially transform input data as a module (a 3DMM-STN) within a convolutional neural network. This is an extension of the original spatial transformer network in that we are able to interpret and normalise 3D pose changes and self-occlusions. The trained localisation part of the network is independently useful since it learns to fit a 3D morphable model to a single image. We show that the localiser can be trained using only simple geometric loss functions on a relatively small dataset yet is able to perform robust normalisation on highly uncontrolled images including occlusion, self-occlusion and large pose changes.