Thinking inside the Convolution for Image Inpainting: Reconstructing Texture via Structure under Global and Local Side

TL;DR

This paper proposes a novel convolutional approach for image inpainting that enhances the reconstruction of textures and structures by addressing information loss during downsampling, leading to superior results across various resolutions.

Contribution

It introduces a statistical normalization strategy for structure and texture feature maps to mitigate information loss in CNN-based inpainting models.

Findings

Outperforms state-of-the-art methods on 256x256 and 512x512 images.

Effective in reconstructing both high-frequency textures and structural details.

Substituting encoders with proposed modules improves inpainting quality.

Abstract

Image inpainting has earned substantial progress, owing to the encoder-and-decoder pipeline, which is benefited from the Convolutional Neural Networks (CNNs) with convolutional downsampling to inpaint the masked regions semantically from the known regions within the encoder, coupled with an upsampling process from the decoder for final inpainting output. Recent studies intuitively identify the high-frequency structure and low-frequency texture to be extracted by CNNs from the encoder, and subsequently for a desirable upsampling recovery. However, the existing arts inevitably overlook the information loss for both structure and texture feature maps during the convolutional downsampling process, hence suffer from a non-ideal upsampling output. In this paper, we systematically answer whether and how the structure and texture feature map can mutually help to alleviate the information loss during the convolutional downsampling. Given the structure and texture feature maps, we adopt the statistical normalization and denormalization strategy for the reconstruction guidance during the convolutional downsampling process. The extensive experimental results validate its advantages to the state-of-the-arts over the images from low-to-high resolutions including 256*256 and 512*512, especially holds by substituting all the encoders by ours. Our code is available at https://github.com/htyjers/ConvInpaint-TSGL