Unbiased Multi-Modality Guidance for Image Inpainting
Yongsheng Yu, Dawei Du, Libo Zhang, Tiejian Luo

TL;DR
This paper introduces an end-to-end multi-modality guided transformer network for image inpainting that leverages multiple auxiliary branches to improve structural guidance, achieving state-of-the-art results efficiently.
Contribution
It proposes a novel multi-modality guided transformer with multi-scale spatial-aware attention, reducing bias and complexity in image inpainting.
Findings
Achieves state-of-the-art inpainting performance on multiple datasets.
Effectively utilizes multi-modal auxiliary information for better structure recovery.
Reduces complexity and bias compared to multi-stage methods.
Abstract
Image inpainting is an ill-posed problem to recover missing or damaged image content based on incomplete images with masks. Previous works usually predict the auxiliary structures (e.g., edges, segmentation and contours) to help fill visually realistic patches in a multi-stage fashion. However, imprecise auxiliary priors may yield biased inpainted results. Besides, it is time-consuming for some methods to be implemented by multiple stages of complex neural networks. To solve this issue, we develop an end-to-end multi-modality guided transformer network, including one inpainting branch and two auxiliary branches for semantic segmentation and edge textures. Within each transformer block, the proposed multi-scale spatial-aware attention module can learn the multi-modal structural features efficiently via auxiliary denormalization. Different from previous methods relying on direct guidance…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Digital Media Forensic Detection
MethodsInpainting
