Image Inpainting with Learnable Bidirectional Attention Maps

TL;DR

This paper introduces a learnable bidirectional attention map module for CNN-based image inpainting, improving handling of irregular holes and producing sharper, more coherent results compared to existing methods.

Contribution

It proposes a novel end-to-end learnable attention mechanism that enhances feature normalization and mask updating for better inpainting of irregular regions.

Findings

Outperforms state-of-the-art methods in qualitative and quantitative tests.

Produces sharper, more coherent, and visually plausible inpainting results.

Effective in handling irregular holes with adaptive feature propagation.

Abstract

Most convolutional network (CNN)-based inpainting methods adopt standard convolution to indistinguishably treat valid pixels and holes, making them limited in handling irregular holes and more likely to generate inpainting results with color discrepancy and blurriness. Partial convolution has been suggested to address this issue, but it adopts handcrafted feature re-normalization, and only considers forward mask-updating. In this paper, we present a learnable attention map module for learning feature renormalization and mask-updating in an end-to-end manner, which is effective in adapting to irregular holes and propagation of convolution layers. Furthermore, learnable reverse attention maps are introduced to allow the decoder of U-Net to concentrate on filling in irregular holes instead of reconstructing both holes and known regions, resulting in our learnable bidirectional attention maps. Qualitative and quantitative experiments show that our method performs favorably against state-of-the-arts in generating sharper, more coherent and visually plausible inpainting results. The source code and pre-trained models will be available.