DreamCom: Finetuning Text-guided Inpainting Model for Image Composition

TL;DR

DreamCom introduces a novel method for image composition by finetuning text-guided inpainting models with reference images, enabling realistic object insertion into backgrounds while preserving object details.

Contribution

The paper proposes a new approach to image composition using finetuned text-guided inpainting models with special tokens and masked attention to improve realism and detail preservation.

Findings

Outperforms existing methods on DreamEditBench and MureCom datasets.

Effectively preserves foreground object details during composition.

Reduces background interference with masked attention mechanisms.

Abstract

The goal of image composition is merging a foreground object into a background image to obtain a realistic composite image. Recently, generative composition methods are built on large pretrained diffusion models, due to their unprecedented image generation ability. However, they are weak in preserving the foreground object details. Inspired by recent text-to-image generation customized for certain object, we propose DreamCom by treating image composition as text-guided image inpainting customized for certain object. Specifically , we finetune pretrained text-guided image inpainting model based on a few reference images containing the same object, during which the text prompt contains a special token associated with this object. Then, given a new background, we can insert this object into the background with the text prompt containing the special token. In practice, the inserted object may be adversely affected by the background, so we propose masked attention mechanisms to avoid negative background interference. Experimental results on DreamEditBench and our contributed MureCom dataset show the outstanding performance of our DreamCom.