Feedback Recorrection Semantic-Based Image Inpainting Under Semi-Supervised Learning

TL;DR

This paper introduces a new image inpainting method that uses feedback between segmentation and inpainting to improve image reconstruction quality.

Contribution

The novel approach introduces a feedback recorrection mechanism between semantic segmentation and inpainting under semi-supervised learning.

Findings

The proposed method achieves a 5.89% reduction in LPIPS and a 0.52% increase in PSNR on the CelebA-HQ dataset.

On the Cityscapes dataset, LPIPS decreases by 6.15% and SSIM increases by 1.58%.

Ablation studies confirm the effectiveness of the feedback recorrection mechanism.

Abstract

Image semantics, by revealing rich structural information, provides crucial guidance for image inpainting. However, current semantic-guided inpainting frameworks generally operate unidirectionally, relying on pre-trained segmentation networks without a feedback mechanism to adapt segmentation dynamically during inpainting. To address this limitation, we propose an innovative inpainting methodology that incorporates semantic segmentation feedback recorrection via semi-supervised learning. Specifically, the fundamental concept involves enabling the initial inpainting network to deliver feedback to the semantic segmentation model, which subsequently refines its predictions by leveraging cross-image semantic consistency. The iteratively corrected semantic segmentation maps serve to direct the inpainting neural network toward improved reconstruction quality, fostering a synergistic interaction that enhances both segmentation accuracy and inpainting performance. Furthermore, a semi-supervised learning strategy is implemented to reduce reliance on ground truth labels and improves generalization by utilizing both labeled and unlabeled datasets. We conduct our methodology on the CelebA-HQnd Cityscapes datasets, employing multiple quantitative metrics including LPIPS, PSNR, and SSIM. Results demonstrate that the proposed algorithm surpasses current methodologies: on CelebA-HQ dataset, it achieves a 5.89% reduction in LPIPS and a 0.52% increase in PSNR, with notable improvements in SSIM; on the Cityscapes dataset, LPIPS decreases by 6.15% and SSIM increases by 1.58%. Ablation studies confirm the effectiveness of the feedback recorrection mechanism. This research provides novel insights into synergistic interactions between segmentation and inpainting, demonstrating that fostering such interactions can substantially improve image processing performance.

Figures