Diffusion-based image inpainting with internal learning

TL;DR

This paper introduces lightweight diffusion models for image inpainting that can be trained on minimal data, achieving competitive results with reduced computational costs across various image types.

Contribution

The paper presents a novel approach to train diffusion models on a single or few images, enabling efficient and effective inpainting for specialized image acquisition modalities.

Findings

Competitive inpainting quality on texture, line drawing, and material images.

Achieves state-of-the-art realism with lower computational load.

Effective for modalities different from standard RGB datasets.

Abstract

Diffusion models are now the undisputed state-of-the-art for image generation and image restoration. However, they require large amounts of computational power for training and inference. In this paper, we propose lightweight diffusion models for image inpainting that can be trained on a single image, or a few images. We show that our approach competes with large state-of-the-art models in specific cases. We also show that training a model on a single image is particularly relevant for image acquisition modality that differ from the RGB images of standard learning databases. We show results in three different contexts: texture images, line drawing images, and materials BRDF, for which we achieve state-of-the-art results in terms of realism, with a computational load that is greatly reduced compared to concurrent methods.