A Theory of Stabilization by Skull Carving

TL;DR

This paper introduces a novel skull stabilization method using neural signed distance fields and isosurface meshing, improving accuracy and robustness in facial motion stabilization across diverse expressions and populations.

Contribution

It presents a new approach leveraging neural SDFs and stable hulls for automatic skull stabilization, outperforming existing methods in complex expression scenarios.

Findings

Enhanced stabilization accuracy on diverse facial expressions

Robustness to sparse and varied expression data

Automatic inclusion of upper teeth in stable hulls

Abstract

Accurate stabilization of facial motion is essential for applications in photoreal avatar construction for 3D games, virtual reality, movies, and training data collection. For the latter, stabilization must work automatically for the general population with people of varying morphology. Distinguishing rigid skull motion from facial expressions is critical since misalignment between skull motion and facial expressions can lead to animation models that are hard to control and can not fit natural motion. Existing methods struggle to work with sparse sets of very different expressions, such as when combining multiple units from the Facial Action Coding System (FACS). Certain approaches are not robust enough, some depend on motion data to find stable points, while others make one-for-all invalid physiological assumptions. In this paper, we leverage recent advances in neural signed distance fields and differentiable isosurface meshing to compute skull stabilization rigid transforms directly on unstructured triangle meshes or point clouds, significantly enhancing accuracy and robustness. We introduce the concept of a stable hull as the surface of the boolean intersection of stabilized scans, analogous to the visual hull in shape-from-silhouette and the photo hull from space carving. This hull resembles a skull overlaid with minimal soft tissue thickness, upper teeth are automatically included. Our skull carving algorithm simultaneously optimizes the stable hull shape and rigid transforms to get accurate stabilization of complex expressions for large diverse sets of people, outperforming existing methods.