MAT: Mask-Aware Transformer for Large Hole Image Inpainting

TL;DR

This paper introduces MAT, a transformer-based model that effectively performs large hole image inpainting by combining transformers and convolutions, achieving high fidelity and diversity in high-resolution images.

Contribution

The paper proposes a novel inpainting-oriented transformer with dynamic masking, enabling efficient high-resolution image inpainting with state-of-the-art results.

Findings

Achieves superior inpainting quality on benchmark datasets.

Effectively models long-range interactions in high-resolution images.

Outperforms existing methods in fidelity and diversity.

Abstract

Recent studies have shown the importance of modeling long-range interactions in the inpainting problem. To achieve this goal, existing approaches exploit either standalone attention techniques or transformers, but usually under a low resolution in consideration of computational cost. In this paper, we present a novel transformer-based model for large hole inpainting, which unifies the merits of transformers and convolutions to efficiently process high-resolution images. We carefully design each component of our framework to guarantee the high fidelity and diversity of recovered images. Specifically, we customize an inpainting-oriented transformer block, where the attention module aggregates non-local information only from partial valid tokens, indicated by a dynamic mask. Extensive experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets. Code is released at https://github.com/fenglinglwb/MAT.