MienCap: Realtime Performance-Based Facial Animation with Live Mood Dynamics
Ye Pan, Ruisi Zhang, Jingying Wang, Nengfu Chen, Yilin Qiu, Yu Ding, Kenny Mitchell

TL;DR
This paper introduces MienCap, a system combining machine learning and traditional techniques to enable real-time and non-real-time performance-based facial animation driven by live mood dynamics, improving realism and efficiency.
Contribution
It presents novel neural network architectures for emotion transfer and blendshape adaptation, achieving geometrically consistent and perceptually valid facial animations in real-time and non-real-time settings.
Findings
Our system outperforms Faceware in expression recognition, intensity, and attractiveness ratings.
The real-time blendshape adaptation network ensures geometric consistency and temporal stability.
The approach enhances animation pipeline efficiency and expression accuracy.
Abstract
Our purpose is to improve performance-based animation which can drive believable 3D stylized characters that are truly perceptual. By combining traditional blendshape animation techniques with multiple machine learning models, we present both non-real time and real time solutions which drive character expressions in a geometrically consistent and perceptually valid way. For the non-real time system, we propose a 3D emotion transfer network makes use of a 2D human image to generate a stylized 3D rig parameters. For the real time system, we propose a blendshape adaption network which generates the character rig parameter motions with geometric consistency and temporally stability. We demonstrate the effectiveness of our system by comparing to a commercial product Faceware. Results reveal that ratings of the recognition, intensity, and attractiveness of expressions depicted for animated…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
