High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation

TL;DR

This paper introduces a deep lighting adaptation model combined with 3D face tracking to improve the realism and robustness of AR/VR avatars under varying lighting conditions, enabling more natural virtual interactions.

Contribution

It presents a novel deep learning lighting model that enhances 3D face tracking for photo-realistic avatars, addressing lighting variability issues in AR/VR applications.

Findings

Improved facial motion transfer accuracy in diverse lighting conditions

Enhanced avatar realism with minimal artifacts

Robustness to pose and expression variations

Abstract

3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven by video, that can minimize uncanny effects, rely on person-specific models. However, existing person-specific photo-realistic 3D models are not robust to lighting, hence their results typically miss subtle facial behaviors and cause artifacts in the avatar. This is a major drawback for the scalability of these models in communication systems (e.g., Messenger, Skype, FaceTime) and AR/VR. This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar. Extensive experimental validation and comparisons to other state-of-the-art methods demonstrate the effectiveness of the proposed framework in real-world scenarios with variability in pose, expression, and illumination. Please visit https://www.youtube.com/watch?v=dtz1LgZR8cc for more results. Our project page can be found at https://www.cs.rochester.edu/u/lchen63.