FreeCompose: Generic Zero-Shot Image Composition with Diffusion Prior

TL;DR

FreeCompose leverages large-scale pre-trained diffusion models to perform generic zero-shot image composition, effectively integrating multiple images into a coherent output without task-specific training.

Contribution

The paper introduces a novel diffusion prior-based method for generic zero-shot image composition, including a mask-guided loss for semantic flexibility.

Findings

Outperforms existing methods in generic image composition tasks

Effective in object removal and multiconcept customization

Utilizes diffusion models' natural boundary detection

Abstract

We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image. Rather than concentrating on specific use cases such as appearance editing (image harmonization) or semantic editing (semantic image composition), we showcase the potential of utilizing the powerful generative prior inherent in large-scale pre-trained diffusion models to accomplish generic image composition applicable to both scenarios. We observe that the pre-trained diffusion models automatically identify simple copy-paste boundary areas as low-density regions during denoising. Building on this insight, we propose to optimize the composed image towards high-density regions guided by the diffusion prior. In addition, we introduce a novel maskguided loss to further enable flexible semantic image composition. Extensive experiments validate the superiority of our approach in achieving generic zero-shot image composition. Additionally, our approach shows promising potential in various tasks, such as object removal and multiconcept customization.