Inpainting at Modern Camera Resolution by Guided PatchMatch with Auto-Curation

TL;DR

This paper introduces a hybrid inpainting framework combining deep learning and traditional methods, achieving high-fidelity 4K image inpainting with a new benchmark dataset and a novel candidate curation process.

Contribution

It presents a new 4K image inpainting benchmark dataset and a hybrid inpainting framework with a novel candidate curation module that outperforms existing methods.

Findings

Overwhelming user preference for the proposed method over baselines.

Quantitative improvements up to 7.4 in metrics over LaMa.

Effective combination of deep models with traditional PatchMatch for high-resolution inpainting.

Abstract

Recently, deep models have established SOTA performance for low-resolution image inpainting, but they lack fidelity at resolutions associated with modern cameras such as 4K or more, and for large holes. We contribute an inpainting benchmark dataset of photos at 4K and above representative of modern sensors. We demonstrate a novel framework that combines deep learning and traditional methods. We use an existing deep inpainting model LaMa to fill the hole plausibly, establish three guide images consisting of structure, segmentation, depth, and apply a multiply-guided PatchMatch to produce eight candidate upsampled inpainted images. Next, we feed all candidate inpaintings through a novel curation module that chooses a good inpainting by column summation on an 8x8 antisymmetric pairwise preference matrix. Our framework's results are overwhelmingly preferred by users over 8 strong baselines, with improvements of quantitative metrics up to 7.4 over the best baseline LaMa, and our technique when paired with 4 different SOTA inpainting backbones improves each such that ours is overwhelmingly preferred by users over a strong super-res baseline.