MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment
Tina Behrouzi, Atefeh Shahroudnejad, Payam Mousavi

TL;DR
MaskRenderer is a real-time, identity-agnostic face reenactment system that leverages 3D modeling and advanced loss functions to improve realism, identity preservation, and occlusion handling, outperforming existing methods especially with large pose changes.
Contribution
The paper introduces MaskRenderer, a novel face reenactment approach combining 3D face modeling, triplet loss, and multi-scale occlusion handling for improved realism and identity preservation.
Findings
Outperforms state-of-the-art models on VoxCeleb1.
Effectively handles large pose changes and occlusions.
Maintains high fidelity and identity accuracy in reenactment.
Abstract
We present a novel end-to-end identity-agnostic face reenactment system, MaskRenderer, that can generate realistic, high fidelity frames in real-time. Although recent face reenactment works have shown promising results, there are still significant challenges such as identity leakage and imitating mouth movements, especially for large pose changes and occluded faces. MaskRenderer tackles these problems by using (i) a 3DMM to model 3D face structure to better handle pose changes, occlusion, and mouth movements compared to 2D representations; (ii) a triplet loss function to embed the cross-reenactment during training for better identity preservation; and (iii) multi-scale occlusion, improving inpainting and restoring missing areas. Comprehensive quantitative and qualitative experiments conducted on the VoxCeleb1 test set, demonstrate that MaskRenderer outperforms state-of-the-art models on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFace recognition and analysis · Facial Nerve Paralysis Treatment and Research · Generative Adversarial Networks and Image Synthesis
MethodsTriplet Loss · Inpainting
