A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization

TL;DR

NeuFace introduces a neural re-parameterized optimization method to generate a large-scale, accurate 3D face mesh video dataset, enhancing 3D face reconstruction and motion modeling.

Contribution

The paper presents NeuFace, a novel neural re-parameterization approach for annotating 3D face meshes in videos, enabling large-scale, accurate datasets for face analysis.

Findings

NeuFace achieves per-view/frame accurate face mesh annotations.

The dataset improves 3D face reconstruction accuracy.

It facilitates learning of 3D facial motion prior.

Abstract

We propose NeuFace, a 3D face mesh pseudo annotation method on videos via neural re-parameterized optimization. Despite the huge progress in 3D face reconstruction methods, generating reliable 3D face labels for in-the-wild dynamic videos remains challenging. Using NeuFace optimization, we annotate the per-view/-frame accurate and consistent face meshes on large-scale face videos, called the NeuFace-dataset. We investigate how neural re-parameterization helps to reconstruct image-aligned facial details on 3D meshes via gradient analysis. By exploiting the naturalness and diversity of 3D faces in our dataset, we demonstrate the usefulness of our dataset for 3D face-related tasks: improving the reconstruction accuracy of an existing 3D face reconstruction model and learning 3D facial motion prior. Code and datasets will be available at https://neuface-dataset.github.io.