Face inpainting with Identity Preserving Latent Diffusion Models

TL;DR

This paper introduces ID-ControlNet, a novel face inpainting method using latent diffusion models that effectively preserves individual identity during occlusion recovery, outperforming existing techniques.

Contribution

The paper proposes a new identity-preserving face inpainting framework based on ControlNet and a specialized training strategy, addressing limitations of prior methods.

Findings

Significantly improves identity preservation over standard diffusion inpainting methods.

Achieves performance comparable to state-of-the-art identity-aware approaches.

Demonstrates robustness across diverse occlusions, poses, and facial variations.

Abstract

Face inpainting techniques recover missing or occluded facial regions in a visually realistic manner, but preserving the identity in the final output remains a fundamental challenge. Identity consistency is crucial for downstream applications such as face recognition, digital forensics, and human-computer interaction, where even subtle identity distortions can significantly degrade performance or trust. Although diffusion-based generative models have recently achieved remarkable progress in image inpainting, they often struggle to faithfully retain individual-specific facial characteristics. On the other hand, existing identity-aware methods typically rely on costly fine-tuning, auxiliary supervision, or exhibit limited robustness to diverse occlusions, poses, and facial variations. To address these limitations, we propose ID-ControlNet, an identity-preserving face inpainting framework built upon latent diffusion models. Based on ControlNet architecture, our approach conditions the diffusion process on facial identity embeddings extracted from a pretrained face recognition network. This design enables reconstruction of occluded facial regions while maintaining global facial coherence and identity fidelity. Furthermore, we introduce an identity consistency and triplet loss training strategy that explicitly enforces alignment between the generated face and the target identity representation. Extensive experiments on CelebA-HQ, FFHQ, and on a new E-Mask dataset demonstrate that ID-ControlNet significantly improves identity preservation over standard diffusion-based inpainting methods, achieving performance comparable to SOTA identity-aware approaches.