VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresence

TL;DR

VOODOO XP is a real-time, 3D-aware head reenactment method that accurately reproduces expressive facial movements from a single portrait and driver video, enabling immersive VR telepresence without calibration.

Contribution

It introduces a novel cross-reenactment architecture with multi-stage self-supervision for highly expressive, view-consistent facial reenactment from minimal input, suitable for VR applications.

Findings

Achieves state-of-the-art expressiveness and likeness preservation.

Operates in real-time without calibration or fine-tuning.

Successfully integrated into an end-to-end VR telepresence system.

Abstract

We introduce VOODOO XP: a 3D-aware one-shot head reenactment method that can generate highly expressive facial expressions from any input driver video and a single 2D portrait. Our solution is real-time, view-consistent, and can be instantly used without calibration or fine-tuning. We demonstrate our solution on a monocular video setting and an end-to-end VR telepresence system for two-way communication. Compared to 2D head reenactment methods, 3D-aware approaches aim to preserve the identity of the subject and ensure view-consistent facial geometry for novel camera poses, which makes them suitable for immersive applications. While various facial disentanglement techniques have been introduced, cutting-edge 3D-aware neural reenactment techniques still lack expressiveness and fail to reproduce complex and fine-scale facial expressions. We present a novel cross-reenactment architecture that directly transfers the driver's facial expressions to transformer blocks of the input source's 3D lifting module. We show that highly effective disentanglement is possible using an innovative multi-stage self-supervision approach, which is based on a coarse-to-fine strategy, combined with an explicit face neutralization and 3D lifted frontalization during its initial training stage. We further integrate our novel head reenactment solution into an accessible high-fidelity VR telepresence system, where any person can instantly build a personalized neural head avatar from any photo and bring it to life using the headset. We demonstrate state-of-the-art performance in terms of expressiveness and likeness preservation on a large set of diverse subjects and capture conditions.