Minutes to Seconds: Speeded-up DDPM-based Image Inpainting with Coarse-to-Fine Sampling

TL;DR

This paper introduces a fast DDPM-based image inpainting method that combines a lightweight model, skip-step sampling, and coarse-to-fine inference to significantly reduce processing time while maintaining high quality.

Contribution

The paper presents a novel combination of speed-up strategies for DDPM-based inpainting, including a lightweight model, skip-step sampling, and coarse-to-fine sampling, achieving about 60 times faster inference.

Findings

Achieves approximately 60 times speedup in image inpainting.

Maintains competitive inpainting quality with faster inference.

Effective on both face and general image datasets.

Abstract

For image inpainting, the existing Denoising Diffusion Probabilistic Model (DDPM) based method i.e. RePaint can produce high-quality images for any inpainting form. It utilizes a pre-trained DDPM as a prior and generates inpainting results by conditioning on the reverse diffusion process, namely denoising process. However, this process is significantly time-consuming. In this paper, we propose an efficient DDPM-based image inpainting method which includes three speed-up strategies. First, we utilize a pre-trained Light-Weight Diffusion Model (LWDM) to reduce the number of parameters. Second, we introduce a skip-step sampling scheme of Denoising Diffusion Implicit Models (DDIM) for the denoising process. Finally, we propose Coarse-to-Fine Sampling (CFS), which speeds up inference by reducing image resolution in the coarse stage and decreasing denoising timesteps in the refinement stage. We conduct extensive experiments on both faces and general-purpose image inpainting tasks, and our method achieves competitive performance with approximately 60 times speedup.