Gradpaint: Gradient-Guided Inpainting with Diffusion Models

TL;DR

GradPaint introduces a gradient-guided inpainting method for diffusion models that enhances global coherence and reduces artifacts by leveraging the model's denoised estimates and a custom loss during inference.

Contribution

It proposes a novel gradient-based guidance mechanism for diffusion models to improve image inpainting quality and coherence without additional training.

Findings

Outperforms current state-of-the-art inpainting methods

Generalizes across various diffusion models and datasets

Reduces artifacts and improves global image coherence

Abstract

Denoising Diffusion Probabilistic Models (DDPMs) have recently achieved remarkable results in conditional and unconditional image generation. The pre-trained models can be adapted without further training to different downstream tasks, by guiding their iterative denoising process at inference time to satisfy additional constraints. For the specific task of image inpainting, the current guiding mechanism relies on copying-and-pasting the known regions from the input image at each denoising step. However, diffusion models are strongly conditioned by the initial random noise, and therefore struggle to harmonize predictions inside the inpainting mask with the real parts of the input image, often producing results with unnatural artifacts. Our method, dubbed GradPaint, steers the generation towards a globally coherent image. At each step in the denoising process, we leverage the model's "denoised image estimation" by calculating a custom loss measuring its coherence with the masked input image. Our guiding mechanism uses the gradient obtained from backpropagating this loss through the diffusion model itself. GradPaint generalizes well to diffusion models trained on various datasets, improving upon current state-of-the-art supervised and unsupervised methods.