VividFace: Real-Time and Realistic Facial Expression Shadowing for Humanoid Robots

TL;DR

VividFace is a system that enables humanoid robots to imitate human facial expressions in real time with high realism, using optimized transfer and inference techniques for lifelike interaction.

Contribution

The paper introduces VividFace, a novel system that achieves real-time, realistic facial expression shadowing for humanoid robots through enhanced transfer models and efficient inference pipelines.

Findings

Produces humanoid facial expressions within 0.05 seconds

Generalizes across diverse facial configurations

Validated through extensive real-world demonstrations

Abstract

Humanoid facial expression shadowing enables robots to realistically imitate human facial expressions in real time, which is critical for lifelike, facially expressive humanoid robots and affective human-robot interaction. Existing progress in humanoid facial expression imitation remains limited, often failing to achieve either real-time performance or realistic expressiveness due to offline video-based inference designs and insufficient ability to capture and transfer subtle expression details. To address these limitations, we present VividFace, a real-time and realistic facial expression shadowing system for humanoid robots. An optimized imitation framework X2CNet++ enhances expressiveness by fine-tuning the human-to-humanoid facial motion transfer module and introducing a feature-adaptation training strategy for better alignment across different image sources. Real-time shadowing is further enabled by a video-stream-compatible inference pipeline and a streamlined workflow based on asynchronous I/O for efficient communication across devices. VividFace produces vivid humanoid faces by mimicking human facial expressions within 0.05 seconds, while generalizing across diverse facial configurations. Extensive real-world demonstrations validate its practical utility. Videos are available at: https://lipzh5.github.io/VividFace/.