Semantic-Guided Two-Stage GAN for Face Inpainting with Hybrid Perceptual Encoding

TL;DR

This paper introduces a semantic-guided two-stage GAN architecture for face inpainting that effectively handles large irregular masks, producing photorealistic and structurally consistent face images by leveraging hierarchical synthesis and multi-modal texture refinement.

Contribution

The proposed method innovatively combines semantic-guided hierarchical synthesis with multi-modal texture refinement, improving face inpainting quality over existing approaches especially for large masks.

Findings

Outperforms state-of-the-art methods on CelebA-HQ and FFHQ datasets.

Achieves better LPIPS, PSNR, and SSIM metrics.

Produces more semantically consistent and visually realistic inpainted faces.

Abstract

Facial Image inpainting aim is to restore the missing or corrupted regions in face images while preserving identity, structural consistency and photorealistic image quality, a task specifically created for photo restoration. Though there are recent lot of advances in deep generative models, existing methods face problems with large irregular masks, often producing blurry textures on the edges of the masked region, semantic inconsistencies, or unconvincing facial structures due to direct pixel level synthesis approach and limited exploitation of facial priors. In this paper we propose a novel architecture, which address these above challenges through semantic-guided hierarchical synthesis. Our approach starts with a method that organizes and synthesizes information based on meaning, followed by refining the texture. This process gives clear insights into the facial structure before we move on to creating detailed images. In the first stage, we blend two techniques: one that focuses on local features with CNNs and global features with Vision Transformers. This helped us create clear and detailed semantic layouts. In the second stage, we use a Multi-Modal Texture Generator to refine these layouts by pulling in information from different scales, ensuring everything looks cohesive and consistent. The architecture naturally handles arbitrary mask configurations through dynamic attention without maskspecific training. Experiment on two datasets CelebA-HQ and FFHQ shows that our model outperforms other state-of-the-art methods, showing improvements in metrics like LPIPS, PSNR, and SSIM. It produces visually striking results with better semantic preservation, in challenging large-area inpainting situations.