Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance

TL;DR

This paper introduces an efficient zero-shot inpainting method using diffusion models that reduces inference cost by avoiding backpropagation through the denoiser, while maintaining high-quality, coherent reconstructions.

Contribution

It proposes a new likelihood surrogate for diffusion-based inpainting that simplifies sampling and significantly cuts inference time without retraining the model.

Findings

Achieves high-quality inpainting comparable to fine-tuned models.

Reduces inference time and memory usage significantly.

Maintains observation consistency in reconstructed images.

Abstract

Diffusion models have emerged as powerful priors for image editing tasks such as inpainting and local modification, where the objective is to generate realistic content that remains consistent with observed regions. In particular, zero-shot approaches that leverage a pretrained diffusion model, without any retraining, have been shown to achieve highly effective reconstructions. However, state-of-the-art zero-shot methods typically rely on a sequence of surrogate likelihood functions, whose scores are used as proxies for the ideal score. This procedure however requires vector-Jacobian products through the denoiser at every reverse step, introducing significant memory and runtime overhead. To address this issue, we propose a new likelihood surrogate that yields simple and efficient to sample Gaussian posterior transitions, sidestepping the backpropagation through the denoiser network. Our extensive experiments show that our method achieves strong observation consistency compared with fine-tuned baselines and produces coherent, high-quality reconstructions, all while significantly reducing inference cost. Code is available at https://github.com/YazidJanati/ding.