Coherent and Multi-modality Image Inpainting via Latent Space Optimization
Lingzhi Pan, Tong Zhang, Bingyuan Chen, Qi Zhou, Wei Ke, Sabine S\"usstrunk, Mathieu Salzmann

TL;DR
This paper introduces PILOT, a latent space optimization method for multi-modal image inpainting that leverages pre-trained diffusion models to produce coherent, high-fidelity inpainted regions aligned with user prompts without additional tuning.
Contribution
The paper proposes a novel latent space optimization approach with semantic centralization and background preservation, enabling effective multi-modal image inpainting using pre-trained diffusion models.
Findings
PILOT outperforms existing methods in coherence and fidelity.
The approach maintains background consistency effectively.
It achieves higher diversity and realism in inpainted images.
Abstract
With the advancements in denoising diffusion probabilistic models (DDPMs), image inpainting has significantly evolved from merely filling information based on nearby regions to generating content conditioned on various prompts such as text, exemplar images, and sketches. However, existing methods, such as model fine-tuning and simple concatenation of latent vectors, often result in generation failures due to overfitting and inconsistency between the inpainted region and the background. In this paper, we argue that the current large diffusion models are sufficiently powerful to generate realistic images without further tuning. Hence, we introduce PILOT (in\textbf{P}ainting v\textbf{I}a \textbf{L}atent \textbf{O}p\textbf{T}imization), an optimization approach grounded on a novel \textit{semantic centralization} and \textit{background preservation loss}. Our method searches latent spaces…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Image Processing Techniques · Image Processing Techniques and Applications · Generative Adversarial Networks and Image Synthesis
MethodsDiffusion · Inpainting
