Unbiased Multi-Modality Guidance for Image Inpainting

TL;DR

This paper introduces an end-to-end multi-modality guided transformer network for image inpainting that leverages multiple auxiliary branches to improve structural guidance, achieving state-of-the-art results efficiently.

Contribution

It proposes a novel multi-modality guided transformer with multi-scale spatial-aware attention, reducing bias and complexity in image inpainting.

Findings

Achieves state-of-the-art inpainting performance on multiple datasets.

Effectively utilizes multi-modal auxiliary information for better structure recovery.

Reduces complexity and bias compared to multi-stage methods.

Abstract

Image inpainting is an ill-posed problem to recover missing or damaged image content based on incomplete images with masks. Previous works usually predict the auxiliary structures (e.g., edges, segmentation and contours) to help fill visually realistic patches in a multi-stage fashion. However, imprecise auxiliary priors may yield biased inpainted results. Besides, it is time-consuming for some methods to be implemented by multiple stages of complex neural networks. To solve this issue, we develop an end-to-end multi-modality guided transformer network, including one inpainting branch and two auxiliary branches for semantic segmentation and edge textures. Within each transformer block, the proposed multi-scale spatial-aware attention module can learn the multi-modal structural features efficiently via auxiliary denormalization. Different from previous methods relying on direct guidance from biased priors, our method enriches semantically consistent context in an image based on discriminative interplay information from multiple modalities. Comprehensive experiments on several challenging image inpainting datasets show that our method achieves state-of-the-art performance to deal with various regular/irregular masks efficiently.