3DFill:Reference-guided Image Inpainting by Self-supervised 3D Image Alignment

TL;DR

3DFill introduces a self-supervised, reference-guided image inpainting method that aligns images using 3D projection and 2D transformation, effectively handling large holes and complex scenes with improved accuracy and speed.

Contribution

The paper presents a novel 3D projection plus 2D transformation alignment approach for reference-guided inpainting, outperforming traditional 2D-only methods.

Findings

Achieves state-of-the-art inpainting performance across wide view shifts.

Faster inference speed compared to existing models.

Effectively handles large holes and complex scenes.

Abstract

Most existing image inpainting algorithms are based on a single view, struggling with large holes or the holes containing complicated scenes. Some reference-guided algorithms fill the hole by referring to another viewpoint image and use 2D image alignment. Due to the camera imaging process, simple 2D transformation is difficult to achieve a satisfactory result. In this paper, we propose 3DFill, a simple and efficient method for reference-guided image inpainting. Given a target image with arbitrary hole regions and a reference image from another viewpoint, the 3DFill first aligns the two images by a two-stage method: 3D projection + 2D transformation, which has better results than 2D image alignment. The 3D projection is an overall alignment between images and the 2D transformation is a local alignment focused on the hole region. The entire process of image alignment is self-supervised. We then fill the hole in the target image with the contents of the aligned image. Finally, we use a conditional generation network to refine the filled image to obtain the inpainting result. 3DFill achieves state-of-the-art performance on image inpainting across a variety of wide view shifts and has a faster inference speed than other inpainting models.