Learning Prior Feature and Attention Enhanced Image Inpainting

TL;DR

This paper introduces a novel image inpainting method that leverages Vision Transformers pre-trained with Masked AutoEncoder to incorporate richer priors and attention mechanisms, significantly improving inpainting quality.

Contribution

The paper proposes integrating MAE pre-training and attention priors from ViT into inpainting models, enhancing their ability to learn long-distance dependencies and richer priors.

Findings

Effective inpainting results on Places2 and FFHQ datasets

Outperforms traditional CNN-based inpainting methods

Demonstrates the benefit of ViT and MAE in image restoration

Abstract

Many recent inpainting works have achieved impressive results by leveraging Deep Neural Networks (DNNs) to model various prior information for image restoration. Unfortunately, the performance of these methods is largely limited by the representation ability of vanilla Convolutional Neural Networks (CNNs) backbones.On the other hand, Vision Transformers (ViT) with self-supervised pre-training have shown great potential for many visual recognition and object detection tasks. A natural question is whether the inpainting task can be greatly benefited from the ViT backbone? However, it is nontrivial to directly replace the new backbones in inpainting networks, as the inpainting is an inverse problem fundamentally different from the recognition tasks. To this end, this paper incorporates the pre-training based Masked AutoEncoder (MAE) into the inpainting model, which enjoys richer informative priors to enhance the inpainting process. Moreover, we propose to use attention priors from MAE to make the inpainting model learn more long-distance dependencies between masked and unmasked regions. Sufficient ablations have been discussed about the inpainting and the self-supervised pre-training models in this paper. Besides, experiments on both Places2 and FFHQ demonstrate the effectiveness of our proposed model. Codes and pre-trained models are released in https://github.com/ewrfcas/MAE-FAR.