HarmonPaint: Harmonized Training-Free Diffusion Inpainting

TL;DR

HarmonPaint is a training-free diffusion inpainting method that uses attention mechanisms and masking strategies to produce coherent, style-harmonized inpainted images without retraining or fine-tuning.

Contribution

It introduces a novel training-free inpainting framework that leverages diffusion models' attention mechanisms for structural and style coherence.

Findings

Effective across diverse scenes and styles

Achieves high-quality, harmonized inpainting without training

Maintains structural fidelity and style transfer

Abstract

Existing inpainting methods often require extensive retraining or fine-tuning to integrate new content seamlessly, yet they struggle to maintain coherence in both structure and style between inpainted regions and the surrounding background. Motivated by these limitations, we introduce HarmonPaint, a training-free inpainting framework that seamlessly integrates with the attention mechanisms of diffusion models to achieve high-quality, harmonized image inpainting without any form of training. By leveraging masking strategies within self-attention, HarmonPaint ensures structural fidelity without model retraining or fine-tuning. Additionally, we exploit intrinsic diffusion model properties to transfer style information from unmasked to masked regions, achieving a harmonious integration of styles. Extensive experiments demonstrate the effectiveness of HarmonPaint across diverse scenes and styles, validating its versatility and performance.