3DRealHead: Few-Shot Detailed Head Avatar

TL;DR

3DRealHead introduces a few-shot 3D head avatar reconstruction method that captures detailed facial expressions using monocular videos and a novel expression control signal, enhancing realism and expressivity.

Contribution

It presents a new few-shot inversion process of a 3D head prior and a novel expression control signal that improves avatar realism and expressivity from limited input data.

Findings

Enables high-fidelity 3D head avatar reconstruction from few images.

Uses mouth features from videos to improve expression accuracy.

Achieves more realistic and expressive avatars than prior methods.

Abstract

The human face is central to communication. For immersive applications, the digital presence of a person should mirror the physical reality, capturing the users idiosyncrasies and detailed facial expressions. However, current 3D head avatar methods often struggle to faithfully reproduce the identity and facial expressions, despite having multi-view data or learned priors. Learning priors that capture the diversity of human appearances, especially, for regions with highly person-specific features, like the mouth and teeth region is challenging as the underlying training data is limited. In addition, many of the avatar methods are purely relying on 3D morphable model-based expression control which strongly limits expressivity. To address these challenges, we are introducing 3DRealHead, a few-shot head avatar reconstruction method with a novel expression control signal that is extracted from a monocular video stream of the subject. Specifically, the subject can take a few pictures of themselves, recover a 3D head avatar and drive it with a consumer-level webcam. The avatar reconstruction is enabled via a novel few-shot inversion process of a 3D human head prior which is represented as a Style U-Net that emits 3D Gaussian primitives which can be rendered under novel views. The prior is learned on the NeRSemble dataset. For animating the avatar, the U-Net is conditioned on 3DMM-based facial expression signals, as well as features of the mouth region extracted from the driving video. These additional mouth features allow us to recover facial expressions that cannot be represented by the 3DMM leading to a higher expressivity and closer resemblance to the physical reality.