Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction

TL;DR

Pixel3DMM leverages vision transformers and foundation model features to improve single-image 3D face reconstruction, achieving higher geometric accuracy across diverse expressions and ethnicities.

Contribution

The paper introduces Pixel3DMM, a novel approach combining vision transformers and foundation model features for enhanced 3D face reconstruction from a single image.

Findings

Outperforms baselines by over 15% in geometric accuracy.

Introduces a new benchmark with diverse expressions and ethnicities.

Employs a novel FLAME fitting optimization for 3DMM parameters.

Abstract

We address the 3D reconstruction of human faces from a single RGB image. To this end, we propose Pixel3DMM, a set of highly-generalized vision transformers which predict per-pixel geometric cues in order to constrain the optimization of a 3D morphable face model (3DMM). We exploit the latent features of the DINO foundation model, and introduce a tailored surface normal and uv-coordinate prediction head. We train our model by registering three high-quality 3D face datasets against the FLAME mesh topology, which results in a total of over 1,000 identities and 976K images. For 3D face reconstruction, we propose a FLAME fitting opitmization that solves for the 3DMM parameters from the uv-coordinate and normal estimates. To evaluate our method, we introduce a new benchmark for single-image face reconstruction, which features high diversity facial expressions, viewing angles, and ethnicities. Crucially, our benchmark is the first to evaluate both posed and neutral facial geometry. Ultimately, our method outperforms the most competitive baselines by over 15% in terms of geometric accuracy for posed facial expressions.