GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding

TL;DR

GeoFill introduces a geometry-aware image inpainting method that uses monocular depth and camera pose estimation to accurately align reference images, significantly improving inpainting quality for complex scenes with large holes.

Contribution

The paper presents a novel geometry-based approach for reference-guided image inpainting that does not rely on scene planarity assumptions, enhancing alignment accuracy.

Findings

Achieves state-of-the-art results on RealEstate10K and MannequinChallenge datasets.

Effectively handles large holes and complex camera motions.

Outperforms previous methods in geometric consistency and inpainting quality.

Abstract

Reference-guided image inpainting restores image pixels by leveraging the content from another single reference image. The primary challenge is how to precisely place the pixels from the reference image into the hole region. Therefore, understanding the 3D geometry that relates pixels between two views is a crucial step towards building a better model. Given the complexity of handling various types of reference images, we focus on the scenario where the images are captured by freely moving the same camera around. Compared to the previous work, we propose a principled approach that does not make heuristic assumptions about the planarity of the scene. We leverage a monocular depth estimate and predict relative pose between cameras, then align the reference image to the target by a differentiable 3D reprojection and a joint optimization of relative pose and depth map scale and offset. Our approach achieves state-of-the-art performance on both RealEstate10K and MannequinChallenge dataset with large baselines, complex geometry and extreme camera motions. We experimentally verify our approach is also better at handling large holes.