Structure First Detail Next: Image Inpainting with Pyramid Generator

TL;DR

This paper introduces a pyramid generator for image inpainting that restores images in a structure-first, detail-later manner, effectively handling large holes and high-resolution images.

Contribution

It proposes a multi-layer pyramid generator with a progressive training scheme, emphasizing structure restoration before details, improving high-resolution inpainting performance.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Effectively restores large-hole and high-resolution images.

Validates the structure-first, detail-next workflow for inpainting.

Abstract

Recent deep generative models have achieved promising performance in image inpainting. However, it is still very challenging for a neural network to generate realistic image details and textures, due to its inherent spectral bias. By our understanding of how artists work, we suggest to adopt a `structure first detail next' workflow for image inpainting. To this end, we propose to build a Pyramid Generator by stacking several sub-generators, where lower-layer sub-generators focus on restoring image structures while the higher-layer sub-generators emphasize image details. Given an input image, it will be gradually restored by going through the entire pyramid in a bottom-up fashion. Particularly, our approach has a learning scheme of progressively increasing hole size, which allows it to restore large-hole images. In addition, our method could fully exploit the benefits of learning with high-resolution images, and hence is suitable for high-resolution image inpainting. Extensive experimental results on benchmark datasets have validated the effectiveness of our approach compared with state-of-the-art methods.