MienCap: Realtime Performance-Based Facial Animation with Live Mood Dynamics

TL;DR

This paper introduces MienCap, a system combining machine learning and traditional techniques to enable real-time and non-real-time performance-based facial animation driven by live mood dynamics, improving realism and efficiency.

Contribution

It presents novel neural network architectures for emotion transfer and blendshape adaptation, achieving geometrically consistent and perceptually valid facial animations in real-time and non-real-time settings.

Findings

Our system outperforms Faceware in expression recognition, intensity, and attractiveness ratings.

The real-time blendshape adaptation network ensures geometric consistency and temporal stability.

The approach enhances animation pipeline efficiency and expression accuracy.

Abstract

Our purpose is to improve performance-based animation which can drive believable 3D stylized characters that are truly perceptual. By combining traditional blendshape animation techniques with multiple machine learning models, we present both non-real time and real time solutions which drive character expressions in a geometrically consistent and perceptually valid way. For the non-real time system, we propose a 3D emotion transfer network makes use of a 2D human image to generate a stylized 3D rig parameters. For the real time system, we propose a blendshape adaption network which generates the character rig parameter motions with geometric consistency and temporally stability. We demonstrate the effectiveness of our system by comparing to a commercial product Faceware. Results reveal that ratings of the recognition, intensity, and attractiveness of expressions depicted for animated characters via our systems are statistically higher than Faceware. Our results may be implemented into the animation pipeline, and provide animators with a system for creating the expressions they wish to use more quickly and accurately.