Learning Sparse Masks for Diffusion-based Image Inpainting

TL;DR

This paper introduces a learned mask generation model for diffusion-based image inpainting that significantly accelerates the process while maintaining high quality, making it more practical for real-world applications like image compression.

Contribution

The authors propose a novel neural network approach to generate inpainting masks efficiently, replacing slow stochastic optimization methods and enabling rapid, high-quality image reconstruction.

Findings

Achieves up to 10,000x faster inpainting process

Maintains competitive image quality with traditional methods

Facilitates practical applications like image compression

Abstract

Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in terms of a learned mask generation model. By emulating the complete inpainting pipeline with two networks for mask generation and neural surrogate inpainting, we obtain a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude. Our findings serve as a basis for making diffusion-based inpainting more attractive for applications such as image compression, where fast encoding is highly desirable.