IMU2Face: Real-time Gesture-driven Facial Reenactment

TL;DR

IMU2Face introduces a real-time facial reenactment system driven by hand gestures captured through IMUs, enabling intuitive control of facial expressions in videos by leveraging common sensors in everyday devices.

Contribution

The paper combines IMU-based gesture tracking with real-time facial reenactment, creating a novel system that uses ubiquitous sensors for intuitive facial expression control.

Findings

Achieves real-time facial reenactment driven by hand gestures.

Utilizes existing IMUs in smartphones and wearables for facial expression control.

Builds on state-of-the-art Face2Face system for face tracking.

Abstract

We present IMU2Face, a gesture-driven facial reenactment system. To this end, we combine recent advances in facial motion capture and inertial measurement units (IMUs) to control the facial expressions of a person in a target video based on intuitive hand gestures. IMUs are omnipresent, since modern smart-phones, smart-watches and drones integrate such sensors, e.g., for changing the orientation of the screen content, counting steps, or for flight stabilization. Face tracking and reenactment is based on the state-of-the-art real-time Face2Face facial reenactment system. Instead of transferring facial expressions from a source to a target actor, we employ an IMU to track the hand gestures of a source actor and use its orientation to modify the target actor's expressions.