Uncertainty-Guided Face Matting for Occlusion-Aware Face Transformation

TL;DR

This paper introduces FaceMat, a trimap-free, uncertainty-aware face matting framework that improves occlusion handling in face filters by estimating high-quality alpha mattes, leveraging a novel training pipeline and a new synthetic dataset.

Contribution

The paper proposes a novel uncertainty-guided, trimap-free face matting method and introduces the CelebAMat dataset for occlusion-aware face processing.

Findings

Outperforms state-of-the-art face matting methods on multiple benchmarks.

Enhances face filter robustness in real-world, occluded scenarios.

Enables real-time, high-quality face compositing without auxiliary inputs.

Abstract

Face filters have become a key element of short-form video content, enabling a wide array of visual effects such as stylization and face swapping. However, their performance often degrades in the presence of occlusions, where objects like hands, hair, or accessories obscure the face. To address this limitation, we introduce the novel task of face matting, which estimates fine-grained alpha mattes to separate occluding elements from facial regions. We further present FaceMat, a trimap-free, uncertainty-aware framework that predicts high-quality alpha mattes under complex occlusions. Our approach leverages a two-stage training pipeline: a teacher model is trained to jointly estimate alpha mattes and per-pixel uncertainty using a negative log-likelihood (NLL) loss, and this uncertainty is then used to guide the student model through spatially adaptive knowledge distillation. This formulation enables the student to focus on ambiguous or occluded regions, improving generalization and preserving semantic consistency. Unlike previous approaches that rely on trimaps or segmentation masks, our framework requires no auxiliary inputs making it well-suited for real-time applications. In addition, we reformulate the matting objective by explicitly treating skin as foreground and occlusions as background, enabling clearer compositing strategies. To support this task, we newly constructed CelebAMat, a large-scale synthetic dataset specifically designed for occlusion-aware face matting. Extensive experiments show that FaceMat outperforms state-of-the-art methods across multiple benchmarks, enhancing the visual quality and robustness of face filters in real-world, unconstrained video scenarios. The source code and CelebAMat dataset are available at https://github.com/hyebin-c/FaceMat.git