Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Space

TL;DR

This paper introduces a self-supervised framework for 3D facial animation that bypasses traditional rigging and blendshape methods, enabling efficient, versatile animation of arbitrary 3D characters from RGBD videos.

Contribution

The novel framework combines hierarchical motion dictionaries and mesh retargeting to animate diverse 3D faces without predefined configurations, streamlining facial animation production.

Findings

Effective in generating realistic 3D facial animations

Works across various character topologies and textures

Reduces time and cost compared to traditional methods

Abstract

Creating realistic 3D facial animation is crucial for various applications in the movie production and gaming industry, especially with the burgeoning demand in the metaverse. However, prevalent methods such as blendshape-based approaches and facial rigging techniques are time-consuming, labor-intensive, and lack standardized configurations, making facial animation production challenging and costly. In this paper, we propose a novel self-supervised framework, Versatile Face Animator, which combines facial motion capture with motion retargeting in an end-to-end manner, eliminating the need for blendshapes or rigs. Our method has the following two main characteristics: 1) we propose an RGBD animation module to learn facial motion from raw RGBD videos by hierarchical motion dictionaries and animate RGBD images rendered from 3D facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D characters regardless of their topology, textures, blendshapes, and rigs; and 2) we introduce a mesh retarget module to utilize RGBD animation to create 3D facial animation by manipulating facial mesh with controller transformations, which are estimated from dense optical flow fields and blended together with geodesic-distance-based weights. Comprehensive experiments demonstrate the effectiveness of our proposed framework in generating impressive 3D facial animation results, highlighting its potential as a promising solution for the cost-effective and efficient production of facial animation in the metaverse.