MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment

TL;DR

MaskRenderer is a real-time, identity-agnostic face reenactment system that leverages 3D modeling and advanced loss functions to improve realism, identity preservation, and occlusion handling, outperforming existing methods especially with large pose changes.

Contribution

The paper introduces MaskRenderer, a novel face reenactment approach combining 3D face modeling, triplet loss, and multi-scale occlusion handling for improved realism and identity preservation.

Findings

Outperforms state-of-the-art models on VoxCeleb1.

Effectively handles large pose changes and occlusions.

Maintains high fidelity and identity accuracy in reenactment.

Abstract

We present a novel end-to-end identity-agnostic face reenactment system, MaskRenderer, that can generate realistic, high fidelity frames in real-time. Although recent face reenactment works have shown promising results, there are still significant challenges such as identity leakage and imitating mouth movements, especially for large pose changes and occluded faces. MaskRenderer tackles these problems by using (i) a 3DMM to model 3D face structure to better handle pose changes, occlusion, and mouth movements compared to 2D representations; (ii) a triplet loss function to embed the cross-reenactment during training for better identity preservation; and (iii) multi-scale occlusion, improving inpainting and restoring missing areas. Comprehensive quantitative and qualitative experiments conducted on the VoxCeleb1 test set, demonstrate that MaskRenderer outperforms state-of-the-art models on unseen faces, especially when the Source and Driving identities are very different.