Reference-based Painterly Inpainting via Diffusion: Crossing the Wild Reference Domain Gap

TL;DR

RefPaint introduces a diffusion-based framework for reference-based painterly inpainting that effectively handles large domain gaps, enabling creative and realistic insertion of objects into artworks.

Contribution

The paper presents a novel diffusion model with a ladder-side branch and masked fusion for large domain gap inpainting, allowing flexible manipulation of semantic and style features.

Findings

RefPaint outperforms existing methods in quality and realism.

The framework enables creative manipulation of style and semantics.

Experiments demonstrate significant improvements in painterly inpainting.

Abstract

Have you ever imagined how it would look if we placed new objects into paintings? For example, what would it look like if we placed a basketball into Claude Monet's ``Water Lilies, Evening Effect''? We propose Reference-based Painterly Inpainting, a novel task that crosses the wild reference domain gap and implants novel objects into artworks. Although previous works have examined reference-based inpainting, they are not designed for large domain discrepancies between the target and the reference, such as inpainting an artistic image using a photorealistic reference. This paper proposes a novel diffusion framework, dubbed RefPaint, to ``inpaint more wildly'' by taking such references with large domain gaps. Built with an image-conditioned diffusion model, we introduce a ladder-side branch and a masked fusion mechanism to work with the inpainting mask. By decomposing the CLIP image embeddings at inference time, one can manipulate the strength of semantic and style information with ease. Experiments demonstrate that our proposed RefPaint framework produces significantly better results than existing methods. Our method enables creative painterly image inpainting with reference objects that would otherwise be difficult to achieve. Project page: https://vita-group.github.io/RefPaint/