Representing 3D Faces with Learnable B-Spline Volumes

TL;DR

CUBE is a novel geometric representation for 3D faces that combines learnable B-spline volumes with neural features, enabling high expressivity and local editing for face reconstruction and registration.

Contribution

It introduces a learnable B-spline volume with high-dimensional control features for improved 3D face modeling and demonstrates its effectiveness in scan registration and monocular reconstruction.

Findings

Achieves state-of-the-art results in 3D scan registration.

Enables local surface editing through control feature updates.

Effectively reconstructs 3D faces from point clouds and images.

Abstract

We present CUBE (Control-based Unified B-spline Encoding), a new geometric representation for human faces that combines B-spline volumes with learned features, and demonstrate its use as a decoder for 3D scan registration and monocular 3D face reconstruction. Unlike existing B-spline representations with 3D control points, CUBE is parametrized by a lattice (e.g., 8 x 8 x 8) of high-dimensional control features, increasing the model's expressivity. These features define a continuous, two-stage mapping from a 3D parametric domain to 3D Euclidean space via an intermediate feature space. First, high-dimensional control features are locally blended using the B-spline bases, yielding a high-dimensional feature vector whose first three values define a 3D base mesh. A small MLP then processes this feature vector to predict a residual displacement from the base shape, yielding the final refined 3D coordinates. To reconstruct 3D surfaces in dense semantic correspondence, CUBE is queried at 3D coordinates sampled from a fixed template mesh. Crucially, CUBE retains the local support property of traditional B-spline representations, enabling local surface editing by updating individual control features. We demonstrate the strengths of this representation by training transformer-based encoders to predict CUBE's control features from unstructured point clouds and monocular images, achieving state-of-the-art scan registration results compared to recent baselines.