Neural Face Skinning for Mesh-agnostic Facial Expression Cloning

TL;DR

This paper introduces a novel facial expression cloning method that combines global and local deformation models, enabling detailed, mesh-agnostic expression transfer with intuitive control and interpretability.

Contribution

It proposes a skinning weight prediction model that localizes global latent influence, improving expression fidelity and adaptability across diverse face meshes.

Findings

Outperforms state-of-the-art in expression fidelity

Achieves accurate deformation transfer on unseen meshes

Enables intuitive editing via FACS-based supervision

Abstract

Accurately retargeting facial expressions to a face mesh while enabling manipulation is a key challenge in facial animation retargeting. Recent deep-learning methods address this by encoding facial expressions into a global latent code, but they often fail to capture fine-grained details in local regions. While some methods improve local accuracy by transferring deformations locally, this often complicates overall control of the facial expression. To address this, we propose a method that combines the strengths of both global and local deformation models. Our approach enables intuitive control and detailed expression cloning across diverse face meshes, regardless of their underlying structures. The core idea is to localize the influence of the global latent code on the target mesh. Our model learns to predict skinning weights for each vertex of the target face mesh through indirect supervision from predefined segmentation labels. These predicted weights localize the global latent code, enabling precise and region-specific deformations even for meshes with unseen shapes. We supervise the latent code using Facial Action Coding System (FACS)-based blendshapes to ensure interpretability and allow straightforward editing of the generated animation. Through extensive experiments, we demonstrate improved performance over state-of-the-art methods in terms of expression fidelity, deformation transfer accuracy, and adaptability across diverse mesh structures.