Reference-Guided Texture and Structure Inference for Image Inpainting

TL;DR

This paper introduces a reference-guided image inpainting approach that leverages a new dataset and a specialized encoder-decoder architecture to improve the filling of complex holes in images, outperforming existing methods.

Contribution

The authors propose a novel reference-guided inpainting method with a dedicated dataset and a feature alignment module, enhancing texture and structure inference for complex scenes.

Findings

Outperforms state-of-the-art methods in complex hole completion

Uses a new dataset of 10K image pairs for training and evaluation

Employs a feature alignment module for better guidance from reference images

Abstract

Existing learning-based image inpainting methods are still in challenge when facing complex semantic environments and diverse hole patterns. The prior information learned from the large scale training data is still insufficient for these situations. Reference images captured covering the same scenes share similar texture and structure priors with the corrupted images, which offers new prospects for the image inpainting tasks. Inspired by this, we first build a benchmark dataset containing 10K pairs of input and reference images for reference-guided inpainting. Then we adopt an encoder-decoder structure to separately infer the texture and structure features of the input image considering their pattern discrepancy of texture and structure during inpainting. A feature alignment module is further designed to refine these features of the input image with the guidance of a reference image. Both quantitative and qualitative evaluations demonstrate the superiority of our method over the state-of-the-art methods in terms of completing complex holes.