Improving Facial Rig Semantics for Tracking and Retargeting

TL;DR

This paper enhances facial rig semantics for more effective tracking and retargeting in VR and gaming by using a unified framework, volumetric morphing, and a novel fine-tuning approach with implicit differentiation.

Contribution

It introduces a fine-tuning method that improves semantic control of facial rigs for better retargeting, leveraging implicit differentiation to adapt to real-world scenarios.

Findings

Calibrated rigs produce desired expressions with animation controls.

Fine-tuning improves semantic meaningfulness of controls for retargeting.

The approach enhances robustness in real-world facial performance retargeting.

Abstract

In this paper, we consider retargeting a tracked facial performance to either another person or to a virtual character in a game or virtual reality (VR) environment. We remove the difficulties associated with identifying and retargeting the semantics of one rig framework to another by utilizing the same framework (3DMM, FLAME, MetaHuman, etc.) for both subjects. Although this does not constrain the choice of framework when retargeting from one person to another, it does force the tracker to use the game/VR character rig when retargeting to a game/VR character. We utilize volumetric morphing in order to fit facial rigs to both performers and targets; in addition, a carefully chosen set of Simon-Says expressions is used to calibrate each rig to the motion signatures of the relevant performer or target. Although a uniform set of Simon-Says expressions can likely be used for all person to person retargeting, we argue that person to game/VR character retargeting benefits from Simon-Says expressions that capture the distinct motion signature of the game/VR character rig. The Simon-Says calibrated rigs tend to produce the desired expressions when exercising animation controls (as expected). Unfortunately, these well-calibrated rigs still lead to undesirable controls when tracking a performance (a well-behaved function can have an arbitrarily ill-conditioned inverse), even though they typically produce acceptable geometry reconstructions. Thus, we propose a fine-tuning approach that modifies the rig used by the tracker in order to promote the output of more semantically meaningful animation controls, facilitating high efficacy retargeting. In order to better address real-world scenarios, the fine-tuning relies on implicit differentiation so that the tracker can be treated as a (potentially non-differentiable) black box.